Air/object determination for biometric sensors

ABSTRACT

A fingerprint sensing apparatus may include a fingerprint sensor system and a control system capable of receiving fingerprint sensor data from the fingerprint sensor system. The control system may be capable of determining fingerprint sensor data blocks for at least a portion of the fingerprint sensor data and of calculating statistical variance values for fingerprint sensor data corresponding to each of the fingerprint sensor data blocks. The control system may be capable of determining, based at least in part the statistical variance values, whether an object is positioned proximate a portion of the fingerprint sensor system.

PRIORITY CLAIM

This application claims priority to, and is a continuation of, U.S.patent application Ser. No. 14/474,163, filed on Aug. 31, 2014 andentitled “AIR/OBJECT DETERMINATION FOR BIOMETRIC SENSORS,” which ishereby incorporated by reference.

TECHNICAL FIELD

This disclosure relates generally to authentication devices and methods,particularly authentication devices and methods applicable to mobiledevices.

DESCRIPTION OF THE RELATED TECHNOLOGY

As mobile devices become more versatile, user authentication becomesincreasingly important. Increasing amounts of personal information maybe stored on and/or accessible by a mobile device. Moreover, mobiledevices are increasingly being used to make purchases and perform othercommercial transactions. Existing authentication methods typicallyinvolve the use of a password or passcode, which may be forgotten by arightful user or used by an unauthorized person. Improved authenticationmethods are desirable.

SUMMARY

The systems, methods and devices of the disclosure each have severalinnovative aspects, no single one of which is solely responsible for thedesirable attributes disclosed herein.

One innovative aspect of the subject matter described in this disclosurecan be implemented in an apparatus, such as a fingerprint sensingapparatus. The fingerprint sensing apparatus may include a fingerprintsensor system and a control system. The control system may include oneor more general purpose single- or multi-chip processors, digital signalprocessors (DSPs), application specific integrated circuits (ASICs),field programmable gate arrays (FPGAs) or other programmable logicdevices, discrete gates or transistor logic, discrete hardwarecomponents, or combinations thereof. The control system may be capableof receiving fingerprint sensor data from the fingerprint sensor systemand of determining, according to the fingerprint sensor data, whether anobject is positioned proximate a portion of the fingerprint sensorsystem.

According to some implementations, if the control system determines thatan object is positioned proximate the portion of the fingerprint sensorsystem, the control system may determine whether the object is a fingeror a non-finger object. As used herein, the term “finger” may refer toany digit, including a finger or a thumb. Accordingly, as used hereinthe term “fingerprint” may refer to a fingerprint or a thumbprint.

In some implementations, if the control system determines that theobject is a finger, the control system may determine whether thefingerprint sensor data includes fingerprint image information of atleast an image quality threshold. In some examples, the control systemmay extract fingerprint features from the fingerprint image informationif fingerprint image information of at least the image quality thresholdis included in the fingerprint sensor data. Extracted fingerprintfeatures may, for example, include fingerprint ridge endings,fingerprint ridge bifurcations, short ridges and/or islands.

In some implementations, the control system may be capable ofdetermining an extracted fingerprint feature quality. The control systemmay be capable of generating a fingerprint template based, at least inpart, on extracted fingerprint features. In some examples, the controlsystem may determine whether to generate a fingerprint template based,at least in part, on the extracted fingerprint feature quality, e.g.,according to an extracted fingerprint feature quality score.

According to some implementations, the control system may be capable ofdetermining a fingerprint template quality. Determining the fingerprinttemplate quality may, for example, involve determining a fingerprinttemplate quality score. The fingerprint template quality score may, forexample, be based on a template matching score, a minutia quality, aminutiae quantity, a feature quality, a feature quantity, imagecontrast, image gradient information, a ridge orientation quality, aridge frequency and/or a ridge flow quality.

In some examples, the control system may be capable of comparing thefingerprint template with a previously-obtained fingerprint template andof determining whether the fingerprint template matches thepreviously-obtained fingerprint template. In some examples, the controlsystem may determine whether to compare the fingerprint template with apreviously-obtained fingerprint template based, at least in part, on thefingerprint template quality of the fingerprint template.

In some implementations, the control system may be capable of updatingthe previously-obtained fingerprint template to include at least one ofthe extracted fingerprint features if the control system determines thatthe fingerprint template matches the previously-obtained fingerprinttemplate. In some such implementations, the control system may determinewhether to update the previously-obtained fingerprint template based, atleast in part, on the fingerprint template quality of the fingerprinttemplate.

According to some implementations, determining whether an object is afinger may involve determining whether the fingerprint sensor dataindicates elements that are within a range of spatial frequencies. Theelements may, for example, be periodic or quasi-periodic elements. Insome examples, determining whether the object is a finger may involvedistinguishing a finger from another body part. In some implementations,determining whether the object is a finger may involve transformingfingerprint sensor data into the frequency domain, performing a textureanalysis, applying a feature detection algorithm and/or applying an edgedetection algorithm.

In some examples, determining whether the object is a finger may involvedetermining that a non-finger object is a stylus. According to some suchimplementations, the control system may be capable of providing a stylusoutput signal indicating whether the non-finger object is a stylus.

In some implementations, the fingerprint sensor system may include anultrasonic sensor array. According to some such implementations, thecontrol system may be capable of assessing an acoustic impedance of oneor more objects proximate the fingerprint sensor system. For example,the control system may be capable of determining whether an object hasan acoustic impedance that is within an acoustic impedance rangecorresponding to that of skin. According to some such implementations,the acoustic impedance range may be from 1.3 MRayls to 2.1 MRayls. Ifthe control system determines that the object has an acoustic impedancethat is within a range corresponding to that of skin, in someimplementations the control system may invoke a process fordistinguishing a finger from another body part.

Other innovative aspects of the subject matter described in thisdisclosure can be implemented in a fingerprint sensing method that mayinvolve receiving fingerprint sensor data from a fingerprint sensorsystem and determining, according to the fingerprint sensor data,whether an object is positioned proximate a portion of the fingerprintsensor system. The method may involve determining, if it is determinedthat an object is positioned proximate the portion of the fingerprintsensor system, whether the object is a finger or a non-finger object.

If it is determined that the object is a finger, the method may involvedetermining whether the fingerprint sensor data includes fingerprintimage information of at least an image quality threshold. The method mayinvolve extracting fingerprint features from the fingerprint imageinformation if it is determined that fingerprint image information of atleast the image quality threshold is included in the fingerprint sensordata.

Some or all of the methods described herein may be performed by one ormore devices according to instructions (e.g., software) stored onnon-transitory media. Such non-transitory media may include memorydevices such as those described herein, including but not limited torandom access memory (RAM) devices, read-only memory (ROM) devices, etc.Accordingly, some innovative aspects of the subject matter described inthis disclosure can be implemented in a non-transitory medium havingsoftware stored thereon.

For example, the software may include instructions for controlling afingerprint sensing apparatus to receive fingerprint sensor data from afingerprint sensor system and to determine, according to the fingerprintsensor data, whether an object is positioned proximate a portion of thefingerprint sensor system. In some examples, the software may includeinstructions for controlling a fingerprint sensing apparatus todetermine, if the fingerprint sensing apparatus determines that anobject is positioned proximate the portion of the fingerprint sensorsystem, whether the object is a finger or a non-finger object.

In some implementations, the software may include instructions forcontrolling the fingerprint sensing apparatus to determine whether thefingerprint sensor data includes fingerprint image information of atleast an image quality threshold. In some examples, the software mayinclude instructions for controlling the fingerprint sensing apparatusto extract fingerprint features from the fingerprint image informationif fingerprint image information of at least the image quality thresholdis included in the fingerprint sensor data.

Other innovative aspects of the subject matter described in thisdisclosure can be implemented in a fingerprint sensing apparatus. Thefingerprint sensing apparatus may include a fingerprint sensor systemand a control system. In some examples, the control system may includeone or more general purpose single- or multi-chip processors, digitalsignal processors (DSPs), application specific integrated circuits(ASICs), field programmable gate arrays (FPGAs) or other programmablelogic devices, discrete gates or transistor logic, discrete hardwarecomponents, or combinations thereof.

The control system may be capable of receiving fingerprint sensor datafrom the fingerprint sensor system, of determining fingerprint sensordata blocks for at least a portion of the fingerprint sensor data, ofcalculating statistical variance values for fingerprint sensor datacorresponding to each of the fingerprint sensor data blocks, and ofdetermining, according to the statistical variance values, whether anobject is positioned proximate a portion of the fingerprint sensorsystem.

According to some implementations, determining whether an object ispositioned proximate a portion of the fingerprint sensor system mayinvolve determining whether the statistical variance values are above orbelow a threshold value. In some examples, the statistical variancevalues may be based, at least in part, on a fingerprint sensor signalvalue or a fingerprint sensor signal gradient.

In some implementations, the control system may be capable of providingan object output signal indicating whether an object is positionedproximate a portion of the fingerprint sensor system. In some suchimplementations, the control system may be capable of determiningwhether an object that is positioned proximate the portion of thefingerprint sensor system is a stylus.

According to some implementations, the fingerprint sensor data blocksmay correspond to blocks of fingerprint sensor pixels of the fingerprintsensor system. The blocks of fingerprint sensor pixels may or may not becontiguous, depending on the particular implementation. In someexamples, the blocks of fingerprint sensor pixels may includesubstantially all fingerprint sensor pixels of the fingerprint sensorsystem. In some implementations, the blocks of fingerprint sensor pixelsmay have at least two different sizes.

In some implementations, the control system may be capable of applyingone or more filters to the fingerprint sensor data before calculatingthe statistical variance values. For example, the one or more filtersmay pass a range of spatial frequencies corresponding to fingerprintfeatures.

Accordingly, in some implementations the control system may be capableof determining whether an object is a finger. In some examples, thecontrol system may extract fingerprint features from the fingerprintsensor data if the control system determines that the object is afinger.

According to some implementations, the fingerprint sensor system mayinclude an ultrasonic sensor array. In some such implementations, thecontrol system may be capable of assessing an acoustic impedance of oneor more objects proximate the fingerprint sensor system. For example,the control system may be capable of determining whether an object hasan acoustic impedance that is within an acoustic impedance rangecorresponding to that of skin. In some implementations, the acousticimpedance range may be from 1.3 MRayls to 2.1 MRayls. However, inalternative implementations, the acoustic impedance range may be adifferent range, such as a narrower range or a broader range. In someimplementations, the control system may be capable of determiningwhether a skin-covered object is a finger or another body part.

Other innovative aspects of the subject matter described in thisdisclosure can be implemented in a fingerprint sensing method. Themethod may involve receiving fingerprint sensor data from a fingerprintsensor system, determining fingerprint sensor data blocks for at least aportion of the fingerprint sensor data, calculating statistical variancevalues for fingerprint sensor data corresponding to each of thefingerprint sensor data blocks and determining, according to thestatistical variance values, whether an object is positioned proximate aportion of the fingerprint sensor system.

For example, the fingerprint sensor data blocks may correspond to blocksof fingerprint sensor pixels. In some implementations, the method mayinvolve determining whether an object is a finger. The method mayinvolve assessing an acoustic impedance of one or more objects proximatethe fingerprint sensor system.

Some innovative aspects of the subject matter described in thisdisclosure can be implemented in a non-transitory medium having softwarestored thereon. For example, the software may include instructions forcontrolling a fingerprint sensing apparatus to receive fingerprintsensor data from a fingerprint sensor system, determine fingerprintsensor data blocks for at least a portion of the fingerprint sensordata, calculate statistical variance values for fingerprint sensor datacorresponding to each of the fingerprint sensor data blocks anddetermine, according to the statistical variance values, whether anobject is positioned proximate a portion of the fingerprint sensorsystem.

For example, the fingerprint sensor data blocks may correspond to blocksof fingerprint sensor pixels. In some implementations, the software mayinclude instructions for controlling the fingerprint sensing apparatusto determine whether an object is a finger. In some instances, thesoftware may include instructions for controlling the fingerprintsensing apparatus to assess an acoustic impedance of one or more objectsproximate the fingerprint sensor system.

Other innovative aspects of the subject matter described in thisdisclosure can be implemented in a fingerprint sensing apparatus. Thefingerprint sensing apparatus may include a fingerprint sensor systemand a control system. The fingerprint sensor system may include anultrasonic sensor array.

In some examples, the control system may include one or more generalpurpose single- or multi-chip processors, digital signal processors(DSPs), application specific integrated circuits (ASICs), fieldprogrammable gate arrays (FPGAs) or other programmable logic devices,discrete gates or transistor logic, discrete hardware components, orcombinations thereof. The control system may be capable of receivingfingerprint sensor data from the fingerprint sensor system, ofdetermining that an object is positioned proximate a portion of thefingerprint sensor system and of determining an acoustic impedance of atleast a portion of the object.

The control system may be capable of determining whether the acousticimpedance is within an acoustic impedance range corresponding to that ofskin and of determining, based at least in part on the acousticimpedance, whether the object is a finger. The acoustic impedance rangemay, for example, be between about 1.3 MRayls and 2.1 MRayls.

In some examples, the control system may be capable of determiningwhether the object is a stylus. In some such implementations, thecontrol system may be capable of providing a stylus output signalindicating whether the object is a stylus.

In some implementations, the control system may be capable ofdistinguishing a finger from another body part. For example,distinguishing a finger from another body part involves distinguishing anose, a cheek, a palm, an elbow, a knuckle or another part of the bodyfrom a finger. The control system may be capable of extractingfingerprint features from the fingerprint sensor data if the controlsystem determines that the object is a finger.

According to some implementations, determining whether the object is afinger may involve detecting patterns associated with fingerprints.Detecting patterns associated with fingerprints may involve a localtexture pattern analysis, a gradient-based pattern analysis, awavelet-based analysis, a frequency domain analysis, or combinationsthereof.

In some implementations, the control system may be capable of applyingone or more filters to the fingerprint sensor data before detectingpatterns associated with fingerprints. The one or more filters may, forexample, pass a range of spatial frequencies corresponding to patternsassociated with fingerprints. The one or more filters may include aband-pass filter, low-pass filter, a median filter, a de-noise filter, aGaussian filter, a wavelet filter and/or a running-average filter.

According to some implementations, detecting patterns associated withfingerprints may involve evaluating fingerprint sensor data blocks forat least a portion of the fingerprint sensor data. The fingerprintsensor data blocks may correspond to blocks of fingerprint sensor pixelsof the fingerprint sensor system. The blocks of fingerprint sensorpixels may or may not be contiguous, depending on the particularimplementation. In some examples, the blocks may include substantiallyall fingerprint sensor pixels that correspond with the object.

In some implementations, determining whether the object is a finger mayinvolve detecting a finger shape. Detecting a finger shape may, forexample, involve detecting a three-dimensional shape.

Other innovative aspects of the subject matter described in thisdisclosure can be implemented in a method of determining whether anobject is a finger. The method may involve receiving fingerprint sensordata from a fingerprint sensor system, determining that an object ispositioned proximate a portion of the fingerprint sensor system anddetermining an acoustic impedance of at least a portion of the object.The method may involve determining whether the acoustic impedance iswithin an acoustic impedance range corresponding to that of skin.

The method may involve determining, based at least in part on theacoustic impedance, whether the object is a finger. However, in someimplementations, determining whether the object is a finger may involvedetecting patterns associated with fingerprints.

Some innovative aspects of the subject matter described in thisdisclosure can be implemented in a non-transitory medium having softwarestored thereon. For example, the software may include instructions forcontrolling a fingerprint sensing apparatus to receive fingerprintsensor data from a fingerprint sensor system, to determine that anobject is positioned proximate a portion of the fingerprint sensorsystem and to determine an acoustic impedance of at least a portion ofthe object. The software may include instructions for controlling thefingerprint sensing apparatus to determine whether the acousticimpedance is within an acoustic impedance range corresponding to that ofskin and to determine, based at least in part on the acoustic impedance,whether the object is a finger.

However, in some implementations, determining whether the object is afinger may involve detecting patterns associated with fingerprints.Detecting patterns associated with fingerprints may, for example,involve a local texture pattern analysis, a gradient-based patternanalysis, a wavelet-based analysis, a frequency domain analysis, orcombinations thereof. In some examples, detecting patterns associatedwith fingerprints may involve evaluating fingerprint sensor data blocksfor at least a portion of the fingerprint sensor data. The fingerprintsensor data blocks may correspond to blocks of fingerprint sensor pixelsof the fingerprint sensor system. According to some implementations, thesoftware may include instructions for controlling the fingerprintsensing apparatus to distinguish a finger from another body part.

BRIEF DESCRIPTION OF THE DRAWINGS

Details of one or more implementations of the subject matter describedin this specification are set forth in the accompanying drawings and thedescription below. Other features, aspects, and advantages will becomeapparent from the description, the drawings, and the claims. Note thatthe relative dimensions of the following figures may not be drawn toscale. Like reference numbers and designations in the various drawingsindicate like elements.

FIG. 1A is a block diagram that shows example components of afingerprint sensing apparatus.

FIG. 1B is a flow diagram that provides examples of fingerprint sensingapparatus operations.

FIG. 1C is a flow diagram that provides additional examples offingerprint sensing apparatus operations.

FIG. 1D is a flow diagram that provides additional examples offingerprint sensing apparatus operations.

FIG. 1E is a flow diagram that provides additional examples offingerprint sensing apparatus operations.

FIG. 1F is a flow diagram that provides additional examples offingerprint sensing apparatus operations.

FIGS. 2A-2L show examples of partial fingerprint images and fingerprintfeatures.

FIG. 3 is a flow diagram that outlines examples of some methods ofupdating a previously-obtained fingerprint template.

FIG. 4A is a flow diagram that provides examples of object detectionoperations.

FIGS. 4B-4D show examples of fingerprint sensor data blocks.

FIG. 4E is a flow diagram that provides additional examples of objectdetection operations.

FIGS. 4F and 4G show examples of air/object determination based, atleast in part, on whether statistical variance values are above or belowa threshold value.

FIG. 4H is a flow diagram that provides additional examples of objectdetection operations.

FIG. 4I is a flow diagram that provides additional examples of objectdetection operations.

FIG. 5 is a block diagram that shows example components of a fingerprintsensing apparatus.

FIG. 6A is a flow diagram that shows example blocks of a method fordetermining whether an object is a finger.

FIG. 6B is a table that provides examples of acoustic impedance valuesfor some common substances.

FIG. 6C is a flow diagram that shows example blocks of an alternativemethod.

FIGS. 7A-7C provide examples of detecting patterns associated withfingerprints and other body parts by evaluating fingerprint sensor datablocks for at least a portion of received fingerprint sensor data.

FIG. 8A shows an example of an exploded view of a touch/fingerprintsensing system.

FIG. 8B shows an exploded view of an alternative example of atouch/fingerprint sensing system.

FIGS. 9A and 9B show examples of system block diagrams illustrating adisplay device that includes a touch/fingerprint sensing system asdescribed herein.

DETAILED DESCRIPTION

The following description is directed to certain implementations for thepurposes of describing the innovative aspects of this disclosure.However, a person having ordinary skill in the art will readilyrecognize that the teachings herein may be applied in a multitude ofdifferent ways. The described implementations may be implemented in anydevice, apparatus, or system that includes a touch/fingerprint sensingsystem. In addition, it is contemplated that the describedimplementations may be included in or associated with a variety ofelectronic devices such as, but not limited to: mobile telephones,multimedia Internet enabled cellular telephones, mobile televisionreceivers, wireless devices, smartphones, Bluetooth® devices, personaldata assistants (PDAs), wireless electronic mail receivers, hand-held orportable computers, netbooks, notebooks, smartbooks, tablets, printers,copiers, scanners, facsimile devices, global positioning system (GPS)receivers/navigators, cameras, digital media players (such as MP3players), camcorders, game consoles, wrist watches, clocks, calculators,television monitors, flat panel displays, electronic reading devices(e.g., e-readers), mobile health devices, computer monitors, autodisplays (including odometer and speedometer displays, etc.), cockpitcontrols and/or displays, camera view displays (such as the display of arear view camera in a vehicle), electronic photographs, electronicbillboards or signs, projectors, architectural structures, microwaves,refrigerators, stereo systems, cassette recorders or players, DVDplayers, CD players, VCRs, radios, portable memory chips, washers,dryers, washer/dryers, parking meters, packaging (such as inelectromechanical systems (EMS) applications includingmicroelectromechanical systems (MEMS) applications, as well as non-EMSapplications), aesthetic structures (such as display of images on apiece of jewelry or clothing) and a variety of EMS devices. Theteachings herein also may be used in applications such as, but notlimited to, electronic switching devices, radio frequency filters,sensors, accelerometers, gyroscopes, motion-sensing devices,magnetometers, inertial components for consumer electronics, parts ofconsumer electronics products, varactors, liquid crystal devices,electrophoretic devices, drive schemes, manufacturing processes andelectronic test equipment. Thus, the teachings are not intended to belimited to the implementations depicted solely in the Figures, butinstead have wide applicability as will be readily apparent to onehaving ordinary skill in the art.

Fingerprint sensors for mobile devices can require extensive computingtime and resources. For example, a frame of fingerprint sensor data maybe acquired in about 50 milliseconds (@20 frames per second). However,it may take about 450 milliseconds to process the fingerprint sensordata fully. Consumers desire fast access times. Moreover, consumers maynot consistently or properly position a digit on a fingerprint scanner.Fingerprint scanning and authentication processes may be startedunintentionally, wasting computing resources and energy. Current systemsgenerate fingerprint templates for all fingerprint sensor data, whetheror not the data includes useable fingerprint information. Currentsystems take the full processing time to process each fingerprint sensordata frame, regardless of data quality. Fingerprint templateinformation, which may include pertinent information about one or morefingerprint features or minutiae, may be stored as a “fingerprinttemplate” in a mobile device. The fingerprint template may be used forenrollment, matching, verification, authentication, and/or otherpurposes by the mobile device.

Various implementations disclosed herein may involve “layered” filteringand/or signal processing for biometric sensors. In some examples,complete processing of fingerprint sensor data may only take place ifeach “layer” or stage concludes successfully. (As noted elsewhereherein, the term “finger” may refer to any digit, including a finger ora thumb, and the term “fingerprint” may refer to a fingerprint or athumbprint.) However, in some implementations at least some processes ofdifferent layers may be performed in parallel. If any layer concludesunsuccessfully, the process may terminate and additional new fingerprintsensor data may be obtained.

In some implementations, an initial screening layer may involvedetermining whether there is an object on or near a fingerprint sensor.This initial or first layer may provide an object/non-object outputsignal indicating whether an object is on or near the fingerprintsensor. The initial or first layer may provide an air/non-air outputsignal indicating whether a material like air or another material is onor near the fingerprint sensor. A second layer may involve determiningwhether the object is a finger (or at least whether the object isfinger-like). The second layer may involve a determination of whetherthe object includes typical fingerprint patterns (e.g., ridges andvalleys). Alternatively, or additionally, the second layer may involvedetermining whether an object has an acoustic impedance that is withinan acoustic impedance range corresponding to that of skin. In someexamples, determining whether the object is a finger may involvedistinguishing a finger from another body part. Determining whether theobject is a finger may involve determining that a non-finger object is astylus. The second layer may provide a finger/non-finger output signalindicating whether a finger is on the fingerprint sensor. The secondlayer may provide an acoustic impedance output signal indicating whetherthe acoustic impedance of an object on the fingerprint sensor is withina predetermined range. The second layer may provide a low-level fraudoutput signal indicating whether an object on the fingerprint sensor maybe a fraud. The second layer may provide a stylus/non-stylus outputsignal indicating whether a tip of a stylus is on the sensor.

A third layer may involve measuring the quality of a fingerprint imageand, in some instances, invoking a routine to improve the quality of theimage. The third layer may provide an image quality output signalindicating an image quality score. The third layer may provide an imagequality output signal indicating whether an image quality is above athreshold. A fourth layer may involve quality scoring of fingerprintfeatures and/or templates, and may involve providing an indication offingerprint matching confidence. The fourth layer may provide and/orstore a feature quality output signal indicating a level of quality fora feature. The fourth layer may provide and/or store a template qualityoutput signal indicating a level of quality of a template. The fourthlayer may provide a recognition confidence level output signalindicating a level of confidence that a template could be matched, orwhether the recognition confidence level is above a predeterminedthreshold. One or more of these output signals may be stored, sent to orotherwise provided to various software applications running on a mobiledevice. For example, the outputs may be provided to one or moreapplication processors, described in more detail with respect to FIGS.9A and 9B.

However, some implementations may include more or fewer layers.Moreover, some implementations disclosed herein may or may not be usedin connection with a “layered” approach, depending on the particularimplementation. For example, some implementations disclosed herein mayinvolve an object/non-object determination that may or may not beimplemented as an initial layer of a multi-layered process, according tothe particular implementation. Some such implementations may involve ablock-based object/non-object determination. For example, some suchimplementations may involve determining fingerprint sensor data blocksand calculating statistical variance values for fingerprint sensor datacorresponding to each of the fingerprint sensor data blocks. Thefingerprint sensor data blocks may correspond to blocks of fingerprintsensor pixels.

Similarly, some implementations disclosed herein may involve afinger/non-finger determination that may or may not be implemented as asecond layer of a multi-layered process, according to the particularimplementation. Some such implementations may involve receivingfingerprint sensor data from a fingerprint sensor system that includesan ultrasonic sensor array. Such implementations may involve determiningan acoustic impedance or a representation of the acoustic impedance ofat least a portion of an object that is positioned proximate thefingerprint sensor system and determining, based at least in part on theacoustic impedance, whether the object is a finger. For example, suchimplementations may involve determining whether the acoustic impedanceis within an acoustic impedance range corresponding to that of skin. Itmay be noted that a value of acoustic impedance may be in non-standardunits such as millivolts or a binary number between, for example, 0 and255 (8 bits), which may serve as a representation of the acousticimpedance in standard units such as MRayls (1E⁶ kg/m²/sec) for means ofcomparison or determining whether an acoustic impedance is within apredetermined range.

Particular implementations of the subject matter described in thisdisclosure can be implemented to realize one or more of the followingpotential advantages. In some implementations, complete processing offingerprint sensor data may only take place if each “layer” or stageconcludes successfully. Therefore, power and computing resources may notbe wasted by processing fingerprint sensor data that does not includefingerprint information of sufficiently high quality. Accordingly, someimplementations disclosed herein may be capable of providing shortaccess times and high accuracy, while minimizing computing time andresources.

FIG. 1A is a block diagram that shows example components of afingerprint sensing apparatus. In this example, the fingerprint sensingapparatus 100 includes a fingerprint sensor system 102 and a controlsystem 104. The fingerprint sensor system 102 may include one or morearrays of fingerprint sensor pixels. The fingerprint sensor system 102may include one or more types of fingerprint sensors, such as opticalsensors, capacitive sensors, etc. The fingerprint sensors may be line orswipe sensors, where a user slides a finger over a surface of a sensor,or area sensors, where a user places a finger on or near a surface of asensor to acquire fingerprint sensor data. In some examples, thefingerprint sensor system 102 may include at least one array ofultrasonic sensors.

The control system 104 may include one or more general purpose single-or multi-chip processors, digital signal processors (DSPs), applicationspecific integrated circuits (ASICs), field programmable gate arrays(FPGAs) or other programmable logic devices, discrete gates ortransistor logic, discrete hardware components, or combinations thereof.The control system 104 also may include (and/or be configured forcommunication with) one or more memory devices, such as one or morerandom access memory (RAM) devices, read-only memory (ROM) devices, etc.The control system 104 may be capable of receiving and processingfingerprint sensor data from the fingerprint sensor system.

FIG. 1B is a flow diagram that provides examples of fingerprint sensingapparatus operations. The blocks of FIG. 1B (and those of other flowdiagrams provided herein) may, for example, be performed by the controlsystem 104 of FIG. 1A or by a similar apparatus. As with other methodsdisclosed herein, the method outlined in FIG. 1B may include more orfewer blocks than indicated. Moreover, the blocks of methods disclosedherein are not necessarily performed in the order indicated.

Here, block 105 involves receiving fingerprint sensor data from afingerprint sensor system. The term “fingerprint sensor data” used hereand elsewhere references data from the fingerprint sensor. However, the“fingerprint sensor data” from the fingerprint sensor will notnecessarily correspond with a fingerprint. For example, in someinstances the fingerprint sensor data may indicate the presence of anobject and/or of air. The fingerprint sensor system may, for example, bethe fingerprint sensor system 102 or a similar apparatus. Accordingly,the blocks of FIG. 1B (and those of other flow diagrams provided herein)may be described with reference to implementations of the control system104 and/or the fingerprint sensor system 102 disclosed herein.

In this example, block 107 involves determining, based at least in parton the fingerprint sensor data, whether an object is positionedproximate a portion of the fingerprint sensor system. Various examplesof object detection are disclosed herein, including implementations fordetermining whether air or an object is proximate a portion of thefingerprint sensor system. In some implementations, the process mayrevert to block 105 if it is determined that no object is positionedproximate at least a portion of the fingerprint sensor system.

Some implementations, e.g., implementations wherein block 105 involvesreceiving “raw” fingerprint sensor data from a fingerprint sensorsystem, may involve pre-processing the fingerprint sensor data. Forexample, some implementations may include one or more pre-processingblocks between block 105 and block 107. Such pre-processing may, forexample, involve or include gain compensation, offset adjustments,background subtraction, de-noise operations, contrast enhancement,scaling, linearization, clipping, dead or low-performing pixelidentification and exclusion, and/or other processes.

If it is determined in block 107 that an object is positioned proximateat least a portion of the fingerprint sensor system, the processcontinues to block 109 in this example. In this implementation, block109 involves determining whether the object is a finger. For example,the control system 104 may be capable of determining whether the objectis a finger or a non-finger object. As noted elsewhere in thisdisclosure, the term “fingerprint sensor data” means data from thefingerprint sensor. Therefore, the “fingerprint sensor data” will notnecessarily correspond with a fingerprint. In some instances thefingerprint sensor data may indicate the presence of an object and/or ofair. Even in cases where the object is a non-finger object, the term“fingerprint sensor data” is used to describe the data (from thenon-finger object) that was received from the fingerprint sensor system.In some implementations, the control system 104 may be capable ofproviding a finger/non-finger output signal indicating whether a fingeris positioned proximate a portion of the fingerprint sensor system.Various examples are described below.

In some implementations wherein the fingerprint sensor system 102includes an ultrasonic sensor array, the control system may be capableof assessing the acoustic impedance of one or more objects proximate thefingerprint sensor system. Determining whether an object is a finger mayinvolve determining whether an object has an acoustic impedance that iswithin an acoustic impedance range corresponding to that of skin (e.g.,between 1.3 MRayls and 2.1 MRayls). As noted elsewhere herein, a valueof acoustic impedance may be in non-standard units such as millivolts ora binary number between, for example, 0 and 255 (8 bits), which mayserve as a representation of the acoustic impedance in standard unitssuch as MRayls for determining whether an acoustic impedance is within apredetermined range. In some implementations, the control system 104 maybe capable of providing an acoustic impedance output signal indicatingwhether the acoustic impedance of an object on or near a portion of thefingerprint sensor is within a predetermined range, such as a rangeincluding skin or other body parts.

In some examples, determining whether the object is a finger may involvedetermining whether the fingerprint sensor data indicates periodic orquasi-periodic elements that are within a range of spatial frequencies.The range of spatial frequencies may correspond with fingerprintfeatures. The range of spatial frequencies may, in some examples, beexpressed in fingerprint line pairs per unit of distance, such as linepairs per millimeter. A fingerprint line pair may, for example, be apair of fingerprint ridges or valleys. In some implementations, a rangeof spatial frequencies that corresponds with fingerprint features may be4-6 fingerprint line pairs per millimeter, or approximately 4-6fingerprint line pairs per millimeter (e.g., 4.5-6 fingerprint linepairs per millimeter, 4.5-5.5 fingerprint line pairs per millimeter,4.0-5.5 fingerprint line pairs per millimeter, etc.). In someimplementations, a range of spatial frequencies that corresponds withfingerprint features may be 3-7 fingerprint line pairs per millimeter,or approximately 3-7 fingerprint line pairs per millimeter (e.g., 3.5-7fingerprint line pairs per millimeter, 3.5-6.5 fingerprint line pairsper millimeter, 3.0-6.5 fingerprint line pairs per millimeter, etc.). Insome implementations, a range of spatial frequencies that correspondswith fingerprint features may be 2-8 fingerprint line pairs permillimeter, or approximately 2-8 fingerprint line pairs per millimeter(e.g., 2.5-8 fingerprint line pairs per millimeter, 2.5-7.5 fingerprintline pairs per millimeter, 2.0-7.5 fingerprint line pairs permillimeter, etc.). In some implementations, the range of spatialfrequencies may be set according to an age and/or finger size of anauthorized user.

According to some implementations, determining whether an object is afinger may involve distinguishing a finger from another body part. Forexample, in some implementations the control system 104 may be capableof distinguishing a finger from another body part, such as a nose, acheek, a palm, an elbow, or a knuckle. In some implementations, forexample, if the control system 104 determines that an object has anacoustic impedance that is within an acoustic impedance rangecorresponding to that of skin, the control system may invoke furtheroperations to distinguish a finger from another body part and/or todistinguish a finger from a non-finger object that has an acousticimpedance that is within an acoustic impedance range corresponding tothat of skin. Such operations may involve transforming fingerprintsensor data into the frequency domain, performing a texture analysis,applying a feature detection algorithm and/or applying an edge detectionalgorithm. However, in alternative implementations, such operations maybe invoked even if the fingerprint sensor system 102 does not include anultrasonic sensor array and/or even if an object's acoustic impedancehas not been determined.

FIG. 1C is a flow diagram that provides additional examples offingerprint sensing apparatus operations. The blocks of FIG. 1C may, forexample, be performed by the control system 104 or by a similarapparatus. In this example, blocks 105 and 107 may be substantially asdescribed above. Block 109 of FIG. 1C also may be substantially asdescribed with reference to FIG. 1B.

However, in this example the finger/non-finger determination of block109 triggers specific subsequent blocks. Here, if it is determined inblock 109 that an object is not a finger, the process continues to block117, in which it is determined whether the non-finger object is astylus. The determination of block 117 may be based upon one or morefactors. For example, the control system 104 may determine whether thenon-finger object is a stylus by evaluating the fingerprint sensor datato determine the object's size, the object's texture, the object's shapeand/or other factors. In this example, the control system 104 is capableof providing a stylus output signal indicating whether a non-fingerobject is a stylus (block 119).

In the example shown in FIG. 1C, if it is determined in block 109 thatan object is a finger, the process continues to block 110, in whichfurther processing occurs. Some examples are described below.

FIG. 1D is a flow diagram that provides additional examples offingerprint sensing apparatus operations. In some implementations, theblocks shown in FIG. 1D may be blocks of an enrollment process. However,at least some blocks of FIG. 1D also may correspond to an authenticationprocess in some examples. The blocks of FIG. 1D may be performed by thecontrol system 104 or by a similar apparatus. In this example, blocks105-107 may be substantially as described above. Block 109 of FIG. 1Calso may be substantially as described with reference to FIGS. 1B and1C.

However, in this example, if it is determined in block 109 that anobject is a finger, it is determined in block 111 whether thefingerprint sensor data includes fingerprint image information of atleast an image quality threshold. If not, no fingerprint features willbe extracted in this implementation, but instead the process reverts toblock 105 in this example. In some implementations, a control system 104may be capable of providing an image quality output signal indicating animage quality score or level, or whether the image quality is above animage quality threshold.

For example, block 111 may involve determining whether the fingerprintsensor data includes fingerprint image information of a sufficientlyhigh image quality to justify the computational burden of extractingfingerprint features. Such a determination may be particularlybeneficial if the computational burden of extracting fingerprintfeatures, subsequent enrollment and/or matching is significantly greaterthan the process of block 111. Accordingly, it may be desirable for theprocess of block 111 to be performed more quickly than the process ofextracting fingerprint features (and/or other processes) and to beperformed with a relatively smaller computational burden. Therefore, insome implementations block 111 may involve evaluating only a portion ofthe fingerprint sensor data that was received in block 105.

According to some implementations, block 111 may involve evaluating thefingerprint image information and making a signal and/or noisedetermination, such as a signal-to-noise ratio determination. Someimplementations of block 111 may involve evaluating image contrastand/or image sharpness. Some implementations may involve evaluating thefingerprint image information to determine a topology of curvescorresponding to ridge and valley structures. Some such implementationsmay involve determining a curve orientation field. In some examples,block 111 may involve evaluating image quality metrics such as theorientation of ridge flow, the clarity of ridge flow and/or ridge flowcontinuity. Some implementations of block 111 may involve evaluating thefingerprint image information for the existence of one or morefingerprint features such as minutia points. Signal-to-noise ratios(SNR) may be determined, for example, by dividing the relative amplitudeof fingerprint sensor data corresponding to ridges of a fingerprint bythe amplitude of the background (such as valleys of a fingerprint orfrom sensor data with no object positioned on the sensor platen). Imagecontrast may be determined, for example, by comparing the magnitude ofthe ridge data with the magnitude of the valley data, such as bygenerating a ratio between one or more ridges and one or more valleys ofthe fingerprint image data. Image sharpness may be determined, forexample, by comparing the magnitudes of ridge data and valley data anddividing by the number of sensor pixels between an edge of a ridge andthe floor of a valley. A topology of curves corresponding to ridges andvalleys may be determined, for example, by constructing a vector fieldassociated with the angular direction of the ridges and valleys disposedon a surface of the sensor platen. A curve orientation field may bedetermined, for example, by constructing a vector field associated withthe curvature of the ridges and valleys (e.g., a relatively straightridge or valley may have a small curvature and a highly curved ridge orvalley such as a whorl may have a high curvature). Image quality metricssuch as the orientation of ridge flow, the clarity of ridge flow, and/orridge flow continuity may be determined, for example, by examining theangular direction, length, continuity and curvature of ridges within thefingerprint sensor data to assess whether a sufficient number offeatures are available to allow successful matching or to identify goodimages for enrollment templates.

Some implementations may involve evaluating the fingerprint imageinformation and determining an image quality score. Some implementationsthat involve determining an image quality score may involve using thescore only for a pass/fail determination in block 111. However, otherimplementations may involve retaining the image quality score andassociating the image quality score with the fingerprint imageinformation. For example some such implementations may involve retainingthe image quality score and associating the image quality score with thefingerprint image information if the image quality score is sufficientlyhigh for a “pass” determination in block 111 (resulting in furtherprocessing, such as proceeding to block 113), but is in a marginal rangeof image quality scores, e.g., less than a predetermined threshold scorefor high image quality.

In some implementations, an image quality threshold for a “pass”determination in block 111 may be different for an enrollment processthan for an authentication process. For example, in some suchimplementations, an image quality threshold for a “pass” determinationin block 111 may be higher for an enrollment process than for anauthentication process. Such differentiation may provide severalpotential advantages. For example, it may be desirable to ensure thatreceived fingerprint image information has a high image quality duringan enrollment process, because the success of accurate futureauthentication processes will depend, at least in part, on the imagequality obtained during the enrollment process. Moreover, a user may bemore tolerant of the time spent during enrollment, as compared to theuser's expectations for an authentication/verification process: ingeneral, users may be less tolerant of delay during day-to-day use of adevice. For example, a user enrolling one or more fingerprints on amobile device may be provided with feedback such as a message or an iconon a display that a finger appears to be dirty when the image quality islow. In another example, a user attempting to be authenticated orverified may receive a message that a finger appears to be dirty or thatthe image quality is low when the image quality is low or an artificialfinger is being used. An authentication process may, for example,validate or verify that a user is who he or she claims to be. Anauthentication process may include an identification sequence, where aparticular user is determined from a set of one or more individuals orauthorized users, such as one or more authorized users of a tabletcomputer or other shared mobile device.

In the example shown in FIG. 1D, if it is determined in block 111 thatthe fingerprint sensor data includes fingerprint image information of atleast an image quality threshold, fingerprint features are extracted inblock 113. The extracted fingerprint features may, for example, includepattern information and/or fingerprint minutiae such as ridge endinginformation, ridge bifurcation information, short ridge information,ridge flow information, island information, spur information, deltainformation, core information, etc.

In this example, block 120 involves generating a fingerprint templatebased, at least in part, on the fingerprint features extracted in block113. For example, block 120 may involve generating fingerprint templateinformation that may include the type, size, location and/or orientationof fingerprint patterns and/or fingerprint minutiae. Here, block 125involves storing the fingerprint template generated in block 120. Thelength of time that the fingerprint template is stored may depend onwhether the process is an enrollment process or an authenticationprocess. If the latter, in some implementations the fingerprint templatemay be deleted after the authentication process is complete.

FIG. 1E is a flow diagram that provides additional examples offingerprint sensing apparatus operations. In some implementations, theblocks shown in FIG. 1E may be blocks of an enrollment process. However,the blocks of FIG. 1E also may correspond to an authentication processin some examples. The blocks of FIG. 1E may be performed by the controlsystem 104 or by a similar apparatus. In this example, blocks 105-113may be substantially as described above.

However, the process outlined in FIG. 1E includes additional qualitythreshold determinations. In this example, after fingerprint featuresare extracted in block 113, it is determined in block 115 whether theextracted fingerprint features are of at least a feature qualitythreshold. If not, no fingerprint template will be created in thisimplementation, but instead the process reverts to block 105 in thisexample. In some implementations, the control system 104 may be capableof providing a feature quality output signal indicating a fingerprintfeature quality, score or level, or whether the feature quality is abovea feature quality threshold.

Block 115 may involve determining an extracted fingerprint featurequality. In some implementations, block 115 may involve determining afingerprint feature quality score, which may be a numeric score in someimplementations. The fingerprint feature quality determination and/orscore may, for example, be an aggregate score based on multipleextracted fingerprint features. In some implementations, substantiallyall extracted fingerprint features will be evaluated in block 115,whereas in other implementations only a subset of the extractedfingerprint features will be evaluated in block 115. In someimplementations, one or more feature quality scores or levels may bestored with the corresponding fingerprint features.

According to some implementations, block 115 may involve evaluating theextracted fingerprint features and making a signal and/or noisedetermination, such as a signal-to-noise ratio determination. Someimplementations of block 115 may involve evaluating feature imagecontrast and/or feature image sharpness. Some implementations of block115 may involve evaluating the orientation of image patterns within thefingerprint image information. Some implementations of block 115 mayinvolve evaluating continuity of image patterns. Some implementations ofblock 115 may involve determining a power spectrum distribution of thefingerprint image information to ascertain, for example, the relativequantity and magnitude of ridge data and valley data. Someimplementations of block 115 may involve evaluating ridge/valley flowquality. Some implementations of block 115 may involve assessing thequality of one or more minutia points in the fingerprint imageinformation, such as determining the contrast of the image in the regionof a minutia point and assigning a value corresponding with the localcontrast to the minutia point (e.g., a minutia point with a higherquality may carry more weight during subsequent matching operations thana minutia point with lower quality). Some implementations of block 115may involve determining the wavelet-level decomposition componentquality of the image information in correspondence with waveletdecomposition, for example, of ridge-valley-ridge features.

In some implementations, block 115 may involve determining whether afeature has a sufficient level to be recognized. For example, some suchimplementations may involve determining a feature recognition confidencelevel and/or a feature recognition confidence level score. The featurerecognition confidence level score may be based, at least in part, onthe fingerprint feature quality determination and/or score. However, insome implementations the feature recognition confidence level score alsomay be based on an image quality score. For example, in someimplementations the feature recognition confidence level score may bebased on a composite score that is based, at least in part, on afingerprint feature quality score and an image quality score. Some suchimplementations may involve applying a weighting factor to one of thescores. In some implementations, a control system may be capable ofproviding a recognition confidence level output signal indicating arecognition confidence level of a fingerprint feature or template,and/or whether the recognition confidence level for the feature ortemplate is above a recognition confidence level threshold.

According to some implementations, block 115 may be an optional process.For example, the extracted fingerprint feature quality determination ofblock 115 may be invoked if the image quality score is sufficiently highfor a “pass” determination in block 111 (resulting in furtherprocessing, such as proceeding to block 113), but is in a marginal rangeof image quality scores, e.g., less than a predetermined threshold scorefor high image quality. If the image quality score is above thepredetermined threshold score for high image quality, block 115 may beomitted in some examples.

In this example, block 120 involves generating a fingerprint templatebased, at least in part, on the fingerprint features extracted in block113. For example, block 120 may involve generating fingerprint templateinformation that may include the type, size, location and/or orientationof fingerprint patterns and/or fingerprint minutiae.

Here, optional block 122 involves determining whether the fingerprinttemplate is of at least a fingerprint template quality threshold. Ifnot, the fingerprint template will not be stored in this implementation,but instead the process reverts to block 105.

Accordingly, block 122 involves determining a fingerprint templatequality in this example. The process of determining a fingerprinttemplate quality may be similar to that of determining fingerprintfeature quality. However, determining a fingerprint template qualityalso may involve determining other factors, such as whether there is asufficient number of high-quality fingerprint features in thefingerprint template, whether high-quality fingerprint features aredistributed over a large enough area for a reliable template match, thesharpness of the image, image contrast, continuity of patterns, etc. Forexample, the fingerprint template quality may be an aggregate of thefingerprint feature quality, such as a count of the fingerprint featuresin the fingerprint template that have a feature quality score assignedto them, or a sum of the feature quality scores for each feature in thetemplate. In this implementation, a high-quality template may have anabundance of high-quality features, whereas a relatively low-qualitytemplate may have only a few features with high quality or a largernumber of features with low quality. During enrollment, for example, auser of a mobile device may be requested to provide one or moreadditional fingerprint images if an initial fingerprint image hasinsufficient feature quality or insufficient template quality to ensureaccurate matching and authentication during subsequent access attempts.During matching, for example, features with the highest quality may beused prior to features with lower quality to determine a match score.Similarly, a template with high quality for one portion of a fingerprintmay be used for matching prior to determining a match from a templatefrom a different portion (overlapping or non-overlapping) of the sameenrolled finger. In some implementations, block 122 may involvedetermining a fingerprint template quality score and determining whetherthe fingerprint template quality score exceeds a threshold. In someimplementations, the fingerprint template quality score may be based, atleast in part, on minutiae quality and/or feature quality. In someimplementations, the fingerprint template quality score may be based, atleast in part, on the quantity of minutiae and/or features in thecorresponding fingerprint image information. In some implementations,the fingerprint template quality score may be based, at least in part,on image contrast and/or image gradient information. In someimplementations, the fingerprint template quality score may be based, atleast in part, on ridge orientation quality, ridge frequency and/orridge flow quality. In some implementations, the fingerprint templatequality score may be based, at least in part, on a template matchingscore. The template matching score, in turn, may be based on the extentof a match with a previously-obtained fingerprint template, e.g., asdescribed below with reference to FIG. 1F. In some implementations, acontrol system may be capable of providing a template quality outputsignal indicating a template quality score or level, and/or whether thetemplate quality is above a template quality threshold.

Here, block 125 involves storing the fingerprint template generated inblock 120. The length of time that the fingerprint template is storedmay depend on whether the process is an enrollment process or anauthentication process. If the process is an authentication process, thefingerprint template may be deleted after the authentication process iscomplete in some implementations. However, as noted below, someimplementations may involve making further use of at least a portion ofa fingerprint template. Some such implementations may involve updating apreviously-obtained fingerprint template. In some implementations, thetemplate quality output signal may be stored with the fingerprinttemplate. In some implementations, one or more feature quality scorescorresponding to one or more fingerprint features may be stored with thefingerprint template. In some implementations, the recognitionconfidence level output signal for a fingerprint feature or template maybe stored with the fingerprint template.

FIG. 1F is a flow diagram that provides additional examples offingerprint sensing apparatus operations. In this example, the blocksshown in FIG. 1F are blocks of an authentication process. The blocks ofFIG. 1F may be performed by the control system 104 or by a similarapparatus. In this example, blocks 105-122 may be substantially asdescribed above.

However, in this implementation, it is determined in block 127 whetherthere is a fingerprint template match. In this example, block 127involves comparing the fingerprint template with a previously-obtainedfingerprint template and determining whether the fingerprint templatematches the previously-obtained fingerprint template. In someimplementations, block 127 may involve determining a template matchingscore that indicates how closely the fingerprint template matches thepreviously-obtained fingerprint template.

If it is determined in block 127 that there is no match (and/or that atemplate matching score is too low), a user will not be authenticated(block 133). Access to a device, such as a device that includes afingerprint sensing apparatus, may be denied. In some implementations,block 133 may involve declining to authorize a purchase or otherfinancial transaction. In this example, the process reverts to block105.

However, if it is determined in block 127 that there is a match (and/orthat a template matching score is sufficiently high), in this examplethe process continues to block 131, in which a user is authenticated.For example, if the authentication process is being performed by afingerprint sensing apparatus of a mobile display device, block 131 mayinvolve allowing a user access to the mobile display device. In someimplementations, block 131 may involve authorizing a purchase or otherfinancial transaction.

Optional block 135 involves determining whether to update thepreviously-obtained fingerprint template. For example, in someimplementations, the previously-obtained fingerprint template may beupdated if it is determined in block 127 that there is a match and ifthe fingerprint template includes at least some features that are notfound in the previously-obtained fingerprint template. In someimplementations, the previously-obtained fingerprint template may beupdated only if the matching score is sufficiently high, indicating ahigh degree of confidence that the fingerprint template and thepreviously-obtained fingerprint template correspond to the same digit ofthe same user. Alternatively, or additionally, the determination ofwhether to update the previously-obtained fingerprint template may bebased, at least in part, on a quality score, such as an image qualityscore, a feature quality score and/or a template quality score.

Accordingly, updating a previously-obtained fingerprint template mayinvolve augmenting the previously-obtained fingerprint template toinclude new features. Alternatively, or additionally, updating apreviously-obtained fingerprint template may involve replacing featuresof the previously-obtained fingerprint template with new featureinformation based on higher-quality images, updating feature locationdata, etc. Such updating features may be desirable for several reasons.For example, a typical user may not understand how to providehigh-quality fingerprint image information. A user may, for example,press too hard on a fingerprint sensor area and distort the naturalpatterns and/or shapes of fingerprint ridges and other features. In someimplementations, fingerprint sensor data may be obtained from arelatively small fingerprint sensor. The fingerprint sensor may, in someinstances, be smaller than a typical human finger. Therefore,fingerprint sensor data from the fingerprint sensor system willcorrespond to only a portion of a total fingerprint.

FIGS. 2A-2L show examples of partial fingerprint images and fingerprintfeatures. In this example, FIGS. 2A-2L are a group of partialfingerprint images 13 that have been obtained from a relatively smallfingerprint sensor with an approximately square active area duringmultiple iterations of a process that includes obtaining fingerprintsensor data, which may be similar to one or more of the processesdescribed above with reference to FIGS. 1B-1F. Each of the partialfingerprint images 13 may correspond with a subset of fingerprint imageinformation that has been obtained during an iteration of the process.According to some such implementations, a subset of fingerprint featurescorresponding to each of the partial fingerprint images 13 may beextracted from each corresponding subset of fingerprint imageinformation and a partial fingerprint template may be generated. Thepartial fingerprint template may, for example, include the types,locations and/or spacing of the fingerprint minutiae 205 g shown in FIG.2B and/or the fingerprint minutiae 205 b shown in FIG. 2H. The partialfingerprint template may correspond with features shown in one or moreof the partial fingerprint images 13.

Some of the subsets of fingerprint image information shown in thepartial fingerprint images 13, such as those shown in FIGS. 2E and 2H,may be deemed to include a sufficient number of fingerprint features foran enrollment process. In some implementations, a subset of fingerprintfeatures corresponding to one or more of the partial fingerprint images13 may have been extracted from a corresponding subset of fingerprintimage information during an enrollment process, and a partialfingerprint template may have been generated and stored. This partialfingerprint template, which would then correspond to an initial versionof a “previously-obtained fingerprint template” disclosed herein, couldbe used as a reference for subsequent authentication procedures, such asthose described above with reference to FIG. 1F. In someimplementations, more than one fingerprint template may be generated andstored for a single finger.

However, in connection with such subsequent authentication procedures,the previously-obtained fingerprint template(s) may be updated. Forexample, suppose that the initial version of the previously-obtainedfingerprint template had been based on the subset of fingerprint imageinformation corresponding with the partial fingerprint image 13 of FIG.2H. In this example, the initial version of the previously-obtainedfingerprint template may include fingerprint features corresponding tothe fingerprint minutiae 205 a-205 d and various other fingerprintfeatures that may be observed in FIG. 2H.

In connection with a subsequent authentication procedure, a newfingerprint template may be based on a subset of fingerprint imageinformation corresponding with the partial fingerprint image 13 of FIG.2E. The new fingerprint template may include fingerprint featurescorresponding to the fingerprint minutiae 205 c-205 f and various otherfingerprint features that may be observed in FIG. 2E. Assuming thatthere is sufficient match between the initial version of thepreviously-obtained fingerprint template and the new fingerprinttemplate, in some implementations the initial version of thepreviously-obtained fingerprint template may be updated to include atleast some of the new fingerprint features, such as the fingerprintfeatures corresponding with the fingerprint minutiae 205 e and 205 f.Some relevant methods and devices are disclosed in paragraphs[0022]-[0055] and the corresponding figures of U.S. patent applicationSer. No. 13/107,635, entitled “Ultrasonic Area-Array Sensor withArea-Image Merging” and filed on May 13, 2011, which material is herebyincorporated by reference.

In a similar fashion, a subsequent authentication procedure may involvegenerating a new fingerprint template based on a subset of fingerprintimage information corresponding with the partial fingerprint image 13 ofFIG. 2B. Assuming that there is sufficient match between the updatedversion of the previously-obtained fingerprint template and the newfingerprint template, in some implementations the updated version of thepreviously-obtained fingerprint template may be updated again to includenew fingerprint features, such as the fingerprint features correspondingwith the fingerprint minutia 205 g. Likewise, a subsequentauthentication procedure may involve generating a new fingerprinttemplate based on a subset of fingerprint image informationcorresponding with the partial fingerprint image 13 of FIG. 2D. Assumingthat there is sufficient match between the updated version of thepreviously-obtained fingerprint template and the new fingerprinttemplate, in some implementations the updated version of thepreviously-obtained fingerprint template may be updated again to includenew fingerprint features, such as the fingerprint features correspondingwith the fingerprint minutiae 205 h and 205 i.

FIG. 3 is a flow diagram that outlines examples of some methods ofupdating a previously-obtained fingerprint template. In this example,method 300 begins with block 305, which involves matching a newfingerprint template with a previously-obtained fingerprint template anda subsequent authentication. In some implementations, block 305 may besimilar to blocks 127 and 131 of FIG. 1F. However, in otherimplementations, block 305 may involve other template matching andauthentication processes. For example, block 305 may involve matching afingerprint template that has been generated without one or more of thequality threshold determinations shown in FIG. 1F.

In this example, block 310 involves determining whether the newfingerprint template includes new and/or higher-quality fingerprintfeatures, as compared to those of the previously-obtained fingerprinttemplate. If so, the previously-obtained fingerprint template may beupdated to include the new and/or higher-quality fingerprint features inblock 315.

According to some such implementations, the updating process may involveaugmenting the previously-obtained fingerprint template to include newfingerprint features, e.g., as described above with reference to FIGS.2A-2L.

Alternatively, or additionally, the updating process of block 315 mayinvolve adapting the previously-obtained fingerprint template. In someimplementations, such adapting may involve replacing fingerprintfeatures of the previously-obtained fingerprint template withhigher-quality corresponding features of the new fingerprint template.

In some implementations, the updating process of block 315 may involveadapting data corresponding to minutiae spacing of thepreviously-obtained fingerprint template. As a child grows, for example,his or her digits will become larger and the spacing between minutiaewill increase. However the types and relative positions of the minutiaemay remain substantially the same. Accordingly, the new fingerprinttemplate may still match a previously-obtained fingerprint template,even though the spacing between minutiae may have increased to somedegree. Block 315 may involve updating the previously-obtainedfingerprint template by changing, scaling, or otherwise adapting datacorresponding to the spacing between at least some of the minutiae. Inthis example, the process ends in block 320. However, someimplementations involve multiple iterations of the blocks shown in FIG.3.

Various examples of object detection are disclosed herein, includingimplementations for determining whether air or an object is proximate aportion of a fingerprint sensor system. The air/object determination maybe useful in determining, for example, whether a fingerprint sensor hasbeen accidentally activated because a mobile device is moved or becausea mobile device contacts a surface, such as the inside of a pocket, etc.

Some implementations may involve making an air/object determinationbefore further processing of possible fingerprint image data. In someexamples, further processing of a possible fingerprint image only takesplace if the air/object determination indicates that image datacorresponding to an object (rather than air) has been acquired. If not,the process may terminate and one or more new fingerprint images may beobtained. Such implementations may provide short access times and highaccuracy, while minimizing computing time and resources.

Various approaches may be used for the air/object determination,depending on the particular implementation. Some approaches may involveevaluating image uniformity. Air proximate the surface of a sensorsystem tends to produce “plain” images that have smaller signalvariations than those of a non-air substance or object. Objects,especially objects with surface patterns, may introduce larger signalvariations. Accordingly, in some implementations, the air/objectdetermination may involve determining statistical variance values for acollection of image data blocks as a measure of higher signal variation(corresponding to an object) or smaller signal variations (correspondingto air). Some implementations may involve block-based variancedeterminations, which may be sufficient for the air/object determinationand which may be performed quickly. For example, in some implementationsblock-based variance determinations sufficient for the air/objectdetermination may be performed in less than one millisecond. However, inalternative implementations such block-based variance determinations maybe performed within a different time interval, e.g., in approximatelyone millisecond or in more than one millisecond.

FIG. 4A is a flow diagram that provides examples of object detectionoperations. At least some blocks of FIG. 4A (like those of other flowdiagrams provided herein) may be performed by the control system 104 ofFIG. 1A or by a similar apparatus. As with other methods disclosedherein, the method outlined in FIG. 4A may include more or fewer blocksthan indicated. Moreover, the blocks are not necessarily performed inthe order indicated.

Here, block 401 involves receiving fingerprint sensor data from afingerprint sensor system. The fingerprint sensor system may, forexample, be the fingerprint sensor system 102 of FIG. 1A or a similarapparatus. Accordingly, some blocks of FIG. 4A are described below withreference to implementations of the control system 104 and/or thefingerprint sensor system 102.

In this example, block 403 involves determining fingerprint sensor datablocks for at least a portion of the fingerprint sensor data. In someimplementations, the process of determining fingerprint sensor datablocks may involve mapping fingerprint sensor pixel position data (e.g.,x-y coordinate data, row/column position data, etc.) to fingerprintsensor data blocks. Accordingly, in some examples the fingerprint sensordata blocks may correspond to blocks of fingerprint sensor pixels of afingerprint sensor system. Some examples are described below withreference to FIGS. 4B-4D.

Here, block 405 involves calculating statistical variance values forfingerprint sensor data corresponding to each of the fingerprint sensordata blocks. Some examples may involve calculating variance according toa naïve algorithm, a two-pass algorithm, an online algorithm, a weightedincremental algorithm, a parallel algorithm, etc. In someimplementations, block 405 may involve calculating standard deviationvalues for fingerprint sensor data corresponding to each of thefingerprint sensor data blocks. In some examples, block 405 may involvecalculating absolute deviation values for fingerprint sensor datacorresponding to each of the fingerprint sensor data blocks. In someimplementations, block 405 may involve calculating the variance ofrelative changes in signal levels between neighboring pixels orneighboring blocks of pixels. In some implementations, block 405 mayinvolve determining the variance of angular orientations of imagegradients for selected pixels or blocks of pixels in the sensor array.

Air on a platen surface tends to produce images that have weakergradients than those of a non-air substance or object. Therefore,alternative implementations may involve calculating a statisticalvariance of fingerprint sensor signal gradient information. In some suchimplementations, the air/object determination may involve determining astandard deviation of fingerprint sensor signal gradient information infingerprint sensor data blocks. The existence of strong gradients maycorrespond to an object and a preponderance of weak gradients maycorrespond to air.

Some implementations, such as implementations wherein block 401 involvesreceiving raw fingerprint sensor data from a fingerprint sensor system,may involve pre-processing the fingerprint sensor data. For example,some implementations may include one or more pre-processing blocksbetween block 401 and block 405 (before and/or after block 403). Suchpre-processing may, for example, involve or include gain compensation,offset adjustments, background subtraction, de-noise operations,contrast enhancement, scaling, linearization, clipping, dead orlow-performing pixel identification and exclusion, and/or otherprocesses. Some filtering examples are described below with reference toFIG. 4E.

According to the implementation shown in FIG. 4A, block 407 involvesdetermining, based at least in part on the statistical variance values,whether an object is positioned proximate a portion of the fingerprintsensor system. In some implementations, a control system may be capableof providing an object output signal indicating whether an object or anon-object (e.g., air) is positioned proximate a portion of thefingerprint sensor system. In some implementations, a control system maybe capable of providing an air/non-air output signal indicating whetherair or a non-air material is positioned proximate a portion of thefingerprint sensor system.

FIGS. 4B-4D show examples of fingerprint sensor data blocks. Each ofFIGS. 4B-4D shows a portion of a fingerprint sensor system, such as thefingerprint sensor system 102 shown in FIG. 1A.

FIG. 4B shows an example of a single fingerprint sensor data block 409.In the example of FIG. 4B, the fingerprint sensor data block 409includes four fingerprint sensor pixels 410. However, a fingerprintsensor data block 409 may include more or fewer than four fingerprintsensor pixels 410. In the implementation shown in FIG. 4C, for example,the fingerprint sensor data block 409 includes 64 fingerprint sensorpixels 410. In other implementations, a fingerprint sensor data block409 may include more or fewer than 64 fingerprint sensor pixels 410. Forexample, in alternative implementations a fingerprint sensor data block409 may include 9, 16, 25, 36, 49, 81, 100, 121, 144, 169, 196, 225,256, 289, 324, 361, 400, 441, 484, 529, 576, 625, 676, 729, 784, 841,900, 961, 1024 or more fingerprint sensor pixels 410. Such fingerprintsensor data blocks 409 may be square or substantially square. Inalternative implementations, a fingerprint sensor data block 409 mayinclude different numbers of fingerprint sensor pixels 410, such asblocks of 4, 8, 16, 32, 64, etc., that are powers of two. Suchfingerprint sensor data blocks 409 may have other shapes, such asrectangular shapes that are not square shapes. In some implementations,the sensor pixels 410 within a block 409 may be contiguous such as in aline or a rectangle. In some implementations, one or more sensor pixels410 within a block 409 may be non-contiguous. In some implementations,the sensor pixels 410 within a block 409 may be sparse and separated byone or more sensor pixels 410 that are not part of the block. In someimplementations, the sensor pixels 410 within a block 409 may be aportion or all of a scan line, one or more adjacent scan lines, one ormore non-adjacent scan lines, or a set of interleaving scan lines.

In some implementations, each of the fingerprint sensor data blocks 409may include the same number of fingerprint sensor pixels 410. However,in alternative implementations, different fingerprint sensor data blocks409 of the same array may include different numbers of fingerprintsensor pixels 410. In one such example, some instances of thefingerprint sensor data blocks 409 may include a first number offingerprint sensor pixels 410 (e.g., 16) and other instances of thefingerprint sensor data blocks 409 may include a second number offingerprint sensor pixels 410 (e.g., 256). Accordingly, in someimplementations the fingerprint sensor data blocks 409 may have at leasttwo different sizes or shapes.

According to some implementations, each of the fingerprint sensor datablocks 409 may be contiguous. In some such implementations, thefingerprint sensor data blocks 409 may include substantially all of thefingerprint sensor pixels 410 of the fingerprint sensor system 102.

However, in some alternative implementations, at least some of thefingerprint sensor data blocks 409 may be non-contiguous. In FIG. 4D,for example, non-contiguous fingerprint sensor data blocks 409 are shownas black squares. The white squares represent areas that are notfingerprint sensor data blocks 409. Accordingly, in this example most ofthe fingerprint sensor pixels 410 of the fingerprint sensor system 102are not included in the fingerprint sensor data blocks 409. Evaluatingonly a portion of the fingerprint sensor pixels 410 of the fingerprintsensor system 102 may be relatively faster and more efficient thanevaluating all of the fingerprint sensor pixels 410, while stillproviding an accurate air/object determination.

FIG. 4E is a flow diagram that provides additional examples of objectdetection operations. At least some blocks of FIG. 4E may be performedby the control system 104 of FIG. 1A or by a similar apparatus. In someimplementations, blocks 401, 403 and 405 may be substantially asdescribed above with reference to FIG. 4A.

However, this example involves applying one or more filters to thefingerprint sensor data before calculating the statistical variancevalues in block 405. In this example, the filters are applied in block402, before determining the fingerprint sensor data blocks in block 403.However, in alternative implementations, the filters may be appliedafter determining the fingerprint sensor blocks. For example, the one ormore filters may include a band-pass filter, a low-pass filter, a medianfilter, a de-noise filter, a Gaussian filter, a wavelet filter and/or arunning-average filter. In some implementations, the filter or filtersmay pass a range of spatial frequencies corresponding to fingerprintfeatures.

As before, block 407 involves determining, based at least in part on thestatistical variance values, whether an object is positioned proximate aportion of the fingerprint sensor system. In the example shown in FIG.4E, block 407 specifically involves determining, based at least in parton whether the statistical variance values are above or below athreshold value, whether an object is positioned proximate a portion ofthe fingerprint sensor system. As noted elsewhere herein, thestatistical variance values may be based, at least in part, on afingerprint sensor signal value or a fingerprint sensor signal gradient.

FIGS. 4F and 4G show examples of air/object determination based, atleast in part, on whether statistical variance values are above or belowa threshold value. In this example, 120 fingerprint sensor data blockshave been evaluated. However, alternative implementations may involveevaluating more or fewer fingerprint sensor data blocks. The fingerprintsensor data blocks may or may not be contiguous and may or may notinclude all of the fingerprint sensor pixels of a fingerprint sensorsystem, depending on the particular implementation.

As shown in FIG. 4F, 40 of the fingerprint sensor data blocks correspondwith fingerprint sensor data having a standard deviation of less than 3,whereas 80 of the fingerprint sensor data blocks correspond withfingerprint sensor data having a standard deviation of more than 3. FIG.4G is a pair of bar graphs that show histograms of underlying standarddeviation data.

As noted above, air proximate the surface of a sensor system tends toproduce “plain” images that have smaller signal variations than those ofa non-air substance or object. Objects, especially objects with surfacepatterns, may introduce larger signal variations. Accordingly, in thisexample, block 407 involves determining that no object is proximate the40 fingerprint sensor data blocks corresponding to a standard deviationof less than 3 and determining that an object is proximate the 80fingerprint sensor data blocks corresponding to a standard deviation ofmore than 3.

In this example shown in FIG. 4G, the standard deviations of blockscorresponding to air are clustered in a range between about 1.5 and 2.7,whereas the standard deviations of blocks corresponding to an object arebroadly distributed in a range from about 7.3 to more than 25. No blockscorrespond with standard deviations in the range from about 1.5 to about7.3. Accordingly, in alternative implementations block 407 may involveother methods of air/object determination that are based, at least inpart, on statistical variance values. For example, such methods could bebased on thresholds other than 3, such as 4 or 5. In someimplementations, the threshold may be adaptively set to increase ordecrease the confidence level when determining an object or air. Inalternative implementations, a dual threshold may be used to providesome hysteresis when changing from an “air” determination to an “object”determination and vice versa, to provide a transition region between alower threshold and an upper threshold. For example, a lower thresholdon the order of 2.5 may be applied to the “air” determination and anupper threshold on the order of 7.5 may be applied to the “object”determination for the block standard deviation data shown in FIG. 4G.

Alternatively, or additionally, other methods of air/objectdetermination may be based, at least in part, on statistical modeling.Such statistical modeling may involve alternative methods ofcharacterizing distributions of standard deviation values or othermetrics related to statistical variance. Some such methods may, forexample, involve an “air” determination based on a cluster or a smallrange of relatively small variance values and/or an “object”determination based on a broader distribution of relatively largervariance values. Some such methods may, for example, involve buildingstatistical models of standard deviation values and/or other metricsrelated to statistical variance for “air” and for “object”. Thedetermination may be “air” if the standard deviation value or othermetrics related to statistical variance matches the “air” model betterthan the “object” model. The determination may be “object” or “non-air”if the standard deviation value or other metrics related to statisticalvariance matches the object model better than the air model.

In some implementations, an “air” or “object” region may be determinedby dividing some or all of the fingerprint image information into aplurality of blocks. Blocks having a pattern or feature may beidentified, such as blocks having a ridge pattern, an edge, a featuregradient, or higher contrast. Blocks without a ridge pattern or otherfeature may be identified, such as blocks with lower gradients, lowersignal or lower contrast (e.g., air). A predetermined or dynamicthreshold level for a quantity such as gradient, signal or contrast maybe established, and blocks below the threshold may be identified as partof an air region, while the remaining blocks may be identified as partof an object region. If a sufficient number of blocks are non-air, anobject on or near the sensor may be ascertained and an object outputsignal may be generated. In some implementations, the air region may beeroded to determine a map or outline of the object region. Furthereroding of the air region (e.g. by elimination of air blocks that arelocated further from the object region and/or by generating additionalblocks near the periphery of the object with a different size, shape orposition and identifying the additional blocks as air or object) mayallow determination of the presence of a stylus or other object on thesurface of a sensor. A stylus/non-stylus output signal may be generatedto indicate if a stylus has been ascertained. In some implementations,the outline of the object region may be used to establish sensor datathat lies in a fingerprint region, and the values of the sensor data inthe fingerprint region may be scaled (e.g., adjust gain and offset) tofurther increase contrast and/or brightness.

FIG. 4H is a flow diagram that provides additional examples of objectdetection operations. At least some blocks of FIG. 4H may be performedby the control system 104 of FIG. 1A or by a similar apparatus. In someimplementations, blocks 401-407 may be substantially as described abovewith reference to FIGS. 4A-4G.

However, in this example, if it is determined in block 407 that anobject is positioned near the fingerprint sensor system, the processcontinues to block 412. Here, block 412 involves determining whether theobject is a stylus. For example, the control system may determinewhether the object is a stylus by evaluating the fingerprint sensor datato determine the object's size, the object's texture, the object'sshape, the object's acoustic impedance level, and/or other factors. Thecontrol system also may be capable of providing an output signalindicating whether the object is a stylus, as shown in block 415.

In some examples, components of a sensor system (which may or may notinclude the fingerprint sensor system, depending on the particularimplementation) may be capable of detecting when a stylus is in contactwith a surface of a display device, such as a cover glass. The sensorsystem may be capable of tracking the motion of the stylus. Positiondata corresponding to the stylus tip may be used, for example, to obtainthe signature of a user and/or to receive stylus input for purposes suchas text input or menu selections.

FIG. 4I is a flow diagram that provides additional examples of objectdetection operations. At least some blocks of FIG. 4I may be performedby the control system 104 of FIG. 1A or by a similar apparatus. In someimplementations, blocks 401-407 may be substantially as described abovewith reference to FIGS. 4A-4G.

However, in this example, if it is determined in block 407 that anobject is positioned near the fingerprint sensor system, the processcontinues to block 417, wherein it is determined whether the object is afinger.

In some implementations wherein the fingerprint sensor system includesan ultrasonic sensor array, a control system may be capable of assessingthe acoustic impedance (or a representation thereof) of one or moreobjects proximate the fingerprint sensor system. Determining whether anobject is a finger may involve determining whether an object has anacoustic impedance that is within an acoustic impedance rangecorresponding to that of skin (e.g., between 1.3 MRayls and 2.1 MRayls).

In some examples, determining whether the object is a finger may involvedetermining whether the fingerprint sensor data indicates periodic orquasi-periodic elements that are within a range of spatial frequencies.The range of spatial frequencies may correspond with fingerprintfeatures. Accordingly, in some implementations a control system may becapable of distinguishing a finger from another body part. Additionalexamples are provided below.

In this example, if it is determined that the object is a finger,fingerprint features are extracted in block 420. Some implementationsalso may involve further processing, such as fingerprint templategeneration, template matching, etc., as described elsewhere herein. Someimplementations may involve one or more quality determinations, such asan image quality determination, a feature quality determination, etc.,as described above.

FIG. 5 is a block diagram that shows example components of a fingerprintsensing apparatus. In this example, the fingerprint sensing apparatus100 includes a fingerprint sensor system 102 and a control system 104.The fingerprint sensor system 102 may include one or more arrays offingerprint sensor pixels. In some examples, the fingerprint sensorsystem 102 may include one or more other types of fingerprint sensors,such as optical sensors, capacitive sensors, etc.

In this implementation, the fingerprint sensor system 102 includes atleast one ultrasonic sensor array 505. In some implementations,components of the ultrasonic sensor array 505 may be similar tocomponents of one of the touch/fingerprint sensing systems 10 that aredescribed below with reference to FIGS. 8A-9B. However, in alternativeimplementations, the ultrasonic sensor array 505 may be configureddifferently. For example, the fingerprint sensor system 102 may or maynot be part of a touch sensor system, depending on the particularimplementation.

The control system 104 may include one or more general purpose single-or multi-chip processors, digital signal processors (DSPs), applicationspecific integrated circuits (ASICs), field programmable gate arrays(FPGAs) or other programmable logic devices, discrete gates ortransistor logic, discrete hardware components, or combinations thereof.The control system 104 also may include (and/or be configured forcommunication with) one or more memory devices, such as one or morerandom access memory (RAM) devices, read-only memory (ROM) devices, etc.The control system 104 may be capable of receiving and processingfingerprint sensor data from the fingerprint sensor system.

FIG. 6A is a flow diagram that shows example blocks of a method fordetermining whether an object is a finger. The blocks of FIG. 6A (andthose of other flow diagrams provided herein) may, for example, beperformed by the control system 104 of FIG. 5 or by a similar apparatus.As with other methods disclosed herein, the method outlined in FIG. 6Amay include more or fewer blocks than indicated. As noted above, theblocks of methods disclosed herein are not necessarily performed in theorder indicated.

Here, block 601 involves receiving fingerprint sensor data from afingerprint sensor system. The fingerprint sensor system may, forexample, be the fingerprint sensor system 102 shown in FIG. 5 or asimilar apparatus. Accordingly, the blocks of FIGS. 6A and 6C may bedescribed with reference to the control system 104 and/or thefingerprint sensor system 102 of FIG. 5.

In this implementation, block 603 involves determining that an object ispositioned proximate a portion of the fingerprint sensor system. Block603 may, in some examples, correspond with blocks 403-407 of FIG. 4Aand/or with blocks 402-407 of FIG. 4E.

In this example, block 605 involves determining an acoustic impedance ofat least a portion of the object. For example, the fingerprint sensordata received in block 601 may include, or may be received with,ultrasonic sensor data from an ultrasonic sensor array such as thatshown in FIG. 5. The control system 104 of FIG. 5 may be capable ofdetermining the acoustic impedance according to the ultrasonic sensordata. As noted elsewhere herein, a value of acoustic impedance may be innon-standard units such as millivolts or a binary number between, forexample, 0 and 255 (8 bits), which may serve as a representation of theacoustic impedance in standard units such as MRayls for determiningwhether an acoustic impedance is within a predetermined range.

Here, block 607 involves determining whether the acoustic impedance iswithin an acoustic impedance range corresponding to that of skin. Inthis example, block 609 involves determining, based at least in part onthe acoustic impedance, whether the object is a finger.

FIG. 6B is a table that provides examples of acoustic impedance valuesfor some common substances. In FIG. 6B, c represents the longitudinalwave velocity (speed of sound) for each corresponding substance. Theacoustic impedance values, represented by Z in FIG. 6B, are expressed inMRayls (1E6 kg/m²s). As noted in FIG. 6B, acoustic impedance is theproduct of c and p, which represents the density of each substance.

As shown in FIG. 6B, the acoustic impedance of soft tissue, such asskin, is similar to the acoustic impedance of fat, water and blood.However, the acoustic impedance of soft tissue is markedly differentfrom that of many other common materials, including bone and aluminum.The acoustic impedance of other metals is generally higher than that ofaluminum. For example, the acoustic impedances of lead, copper,molybdenum and tungsten are 24.6, 44.6, 63.1 and 101 MRayls,respectively. The acoustic impedance of glass is in the range of 12-16MRayls.

Accordingly, if an object positioned near a fingerprint sensor systemhas an acoustic impedance that is within an acoustic impedance rangecorresponding to that of skin, there is a reasonable likelihood that theobject is a finger or another body part. In some implementations, theacoustic impedance range may be from about 1.3 MRayls to about 2.1MRayls, depending in part on the toughness of the skin, the amount ofadipose tissue or fat, the amount of muscle tissue, the prevalence ofblood and other biological materials at or near the surface of the body,and to a certain extent on the operating frequency. Alternatively, theacoustic impedance range may be a slightly different range, e.g., fromabout 1.7 MRayls to about 2.1 MRayls.

FIG. 6C is a flow diagram that shows example blocks of an alternativemethod. The blocks of FIG. 6C (and those of other flow diagrams providedherein) may, for example, be performed by the control system 104 of FIG.5 or by a similar apparatus. In some implementations, blocks 601-609 maybe performed substantially as described above with reference to FIG. 6A.

In this example, if it is determined in block 607 that the acousticimpedance is not within a range corresponding to that of skin, theprocess continues to block 611. In this implementation, block 611involves determining whether the object is a stylus. Block 611 may, forexample, be performed substantially as described above with reference toblock 117 of FIG. 1C. After determining in block 611 whether the objectis a stylus, in this example an output signal is provided that indicateswhether or not the object is a stylus (block 613).

However, if it is determined in block 607 that the acoustic impedance iswithin a range corresponding to that of skin, in this example theprocess continues to block 609. Here, block 609 involves determiningwhether the object is a finger.

As noted above, if an object positioned near a fingerprint sensor systemhas an acoustic impedance that is within an acoustic impedance rangecorresponding to that of skin, there is a reasonable likelihood that theobject is a finger or another body part. Accordingly, in someimplementations, block 609 may involve determining whether askin-covered object is a finger or another body part. For example, insome such implementations, distinguishing a finger from another bodypart may involve distinguishing a nose, a cheek, a palm, an elbow, aknuckle or another part of the body from a finger. In someimplementations, a control system may be capable of providing a bodypart output signal indicating whether a body part is positionedproximate a portion of the fingerprint sensor system. In someimplementations, a control system may be capable of providing alow-level fraud output signal indicating whether an object positionedproximate a portion of the fingerprint sensor system is not a finger ora suitable body part when, for example, a fingerprint or otheridentifiable body feature is expected during enrollment, verification,or identification.

According to some such implementations, determining whether the objectis a finger involves detecting patterns associated with fingerprints.Some examples are described below with reference to FIGS. 7A-7C.

If it is determined in block 609 that the object is a finger, in thisexample fingerprint features are extracted in block 615. Block 615 maybe substantially similar to block 113 of FIGS. 1D-1F. In someimplementations, fingerprint features may be extracted only if it isdetermined that the fingerprint sensor data includes fingerprint imageinformation of at least an image quality threshold, e.g., as describedabove with reference to block 111 of FIGS. 1D-1F. Moreover, someimplementations may involve further processing that involves theextracted fingerprint features, e.g., as described above with referenceto FIGS. 1D-3.

FIGS. 7A-7C provide examples of detecting patterns associated withfingerprints and other body parts by evaluating fingerprint sensor datablocks for at least a portion of received fingerprint sensor data. FIG.7A shows examples of images corresponding to fingerprint sensor data.Images 705-720 are fingerprint images, images 725-740 are images ofportions of a nose, a cheek, a palm and an elbow, respectively, andimage 745 is an image of air.

In this example, each of the images of FIG. 7A corresponds to afingerprint sensor data block. Here, each fingerprint sensor data blockcorresponds to a block of fingerprint sensor pixels of a fingerprintsensor system. In this example, each image of FIG. 7A corresponds to a64×64 block of fingerprint sensor pixels. Accordingly, the numbers alongthe x and y axes of each image correspond to fingerprint sensor pixelsof each fingerprint sensor data block. In some implementations, theblocks of fingerprint sensor pixels used for the analysis may becontiguous. In some such implementations, the blocks may include all, orsubstantially all, fingerprint sensor pixels that correspond with anobject. However, in alternative implementations, the blocks offingerprint sensor pixels may be non-contiguous and may not include all,or substantially all, fingerprint sensor pixels that correspond with anobject.

As shown in FIG. 7A, fingerprints may have oval-shaped edges as well asother structured patterns, such as ridges and valleys, whorls,bifurcations, etc. However, the images 725-740, which correspond toother body parts, lack some or all of these shapes and/or patterns.Therefore, in some implementations, the finger/non-finger determinationof block 609 may involve an image-based analysis to detect shapes and/orpatterns associated with fingerprints or other parts of the body.

Accordingly, determining whether the object is a finger may involvedetecting a finger shape. Detecting a finger shape may involve detectinga three-dimensional shape, e.g., assessing the object's texture and/ortopology. Therefore, some implementations may involve a texture analysisto determine whether the object includes fingerprint-like textures orstructures. Such implementations may involve applying one or morefeature detection algorithms, such as edge detection algorithms. Suchassessments also may be useful in determining whether a non-fingerobject is a stylus, as most styli tend to be smooth and small with sharpboundaries.

Accordingly, some implementations may involve a local texture patternanalysis, a gradient-based pattern analysis, a wavelet-based analysis, afrequency domain analysis, or combinations thereof. Some implementationsmay involve applying one or more filters to the fingerprint sensor databefore detecting patterns associated with fingerprints. In someimplementations, the one or more filters may pass a range of spatialfrequencies corresponding to patterns associated with fingerprints. Theone or more filters may include a band-pass filter, a low-pass filter, amedian filter, a de-noise filter, a Gaussian filter, a wavelet filterand/or a running-average filter.

The ridges and valleys of fingerprints, though all are different, tendto be within a predictable range of spatial frequencies. Accordingly, insome implementations, the finger/non-finger determination may involvedetermining whether an object includes a pattern having periodic orquasi-periodic elements. The finger/non-finger determination of block609 may involve detecting periodic or quasi-periodic elements withinthis range of spatial frequencies in the fingerprint sensor data. Insome implementations, the finger/non-finger determination of block 609may involve detecting a distribution of the orientations of periodic orquasi-periodic elements within this range of spatial frequencies.

Some implementations may involve transforming data into the spatialfrequency domain for analysis. Ridge-valley-type patterns can produce apair of peaks or rings in the frequency domain. For images broken intoan array 64×64 pixel blocks, such as the images shown in FIG. 7A, theridge distance of fingerprint images at 500 dots per inch (dpi) isroughly 6 to 8 pixels, which can lead to a specific range of peaks/ringsfor fingerprints in the frequency domain. This is a further example of arange of spatial frequencies for a finger/non-finger determination.

FIG. 7B shows examples of transforming the corresponding images of FIG.7A into the frequency domain. The frequency domain representation ofeach of the fingerprint images includes a characteristic type of peaksor rings. As shown in FIG. 7B, the presence or absence of such featurescan determine whether a human body part for which an image has beenobtained corresponds to a fingerprint, a nose print, an elbow print,etc. The higher-energy (higher-intensity) areas of FIG. 7B arerepresented as lighter areas, whereas the lower-energy areas are darker.FIG. 7C is a series of bar graphs that show examples of energy(intensity) distributions on a log scale for each of the frequencydomain transforms of FIG. 7B, with the x-axis in pixel units in thefrequency domain.

The frequency domain representations of FIG. 7B that correspond tofingerprint images share similar characteristics. For example, thefrequency domain representations corresponding to images 705-720 allinclude a pair of light, high-energy areas. Referring to FIG. 7C, theenergy distributions for each of the frequency domain transforms thatcorrespond to fingerprint images reach a peak at radius values ofapproximately 6-8 units.

However, there are some differences within these frequency domainrepresentations of fingerprint images. The frequency domainrepresentations corresponding to images 705 and 720, which includefingerprint ridges and valleys having relatively little curvature,include a pair of high-energy areas ((750 a, 750 b) and (750 g, 750 h))that are more localized and relatively more circular in shape. Incontrast, the frequency domain representations corresponding to images710 and 715, which represent arcuate fingerprint ridges and valleyshaving relatively more curvature, include pairs of high-energy areas((750 c, 750 d) and (750 e, 750 f)) that are “smeared” or distributedwithin a range of radii. It may be observed that predominantly verticalridges and valleys as in fingerprint image 705 result in a pair ofhigh-energy areas 750 a, 750 b that are predominantly horizontal (e.g.the direction of highest gradient), whereas the predominantly horizontalridges and valleys as in fingerprint image 720 result in a pair ofhigh-energy areas 750 g, 750 h that are predominantly vertical. Theparticular orientation of the high-energy areas can provide anindication of the direction of ridge flow within a portion of afingerprint image.

The frequency domain representations corresponding to images 725, 730and 740, which are nose, cheek and elbow images, respectively, do notinclude a pair of light, high-energy areas. Instead, the high-energyareas 750 i, 750 j and 750 m are distributed more uniformly. Thecorresponding graphs of FIG. 7C show relatively “flatter” energydistributions over the same range of radii, as compared to the graphscorresponding to fingerprint images.

The frequency domain representation corresponding to image 735, which isa palm image, does show a pair of light, high-energy areas (750 k, 750l). Accordingly, palm images may be relatively more difficult todifferentiate from fingerprint images than images of other body parts.In this example, the pair of light, high-energy areas (750 k, 750 l) iseven more localized than the pairs of high-energy areas ((750 a, 750 b)and (750 g, 750 h)) that may correspond to fingerprint images. Thecorresponding energy distribution graph in FIG. 7C reaches a peak at aradius value of about 5 units (smaller spatial frequency correspondingto larger distances between pronounced features), as compared to thepeaks in the range of 6-8 units for the fingerprint images (largerspatial frequency corresponding to smaller distances between pronouncedfeatures). While frequency domain representations of air (e.g., 750 n)may show some features due in part to row, column or pixel variationswithin the sensor, the intensity of the distributions may be appreciablyreduced compared to images of fingers or other body parts.

If a sufficient number of blocks are analyzed, relatively more “smeared”pairs of high-energy areas, corresponding to arcuate fingerprint ridgesand valleys should be detected if the images are actually fingerprintimages. Therefore, by assessing a sufficient number of blocks,differences between a palm image and a fingerprint image should becomeapparent.

FIG. 8A shows an example of an exploded view of a touch/fingerprintsensing system. In this example, the touch/fingerprint sensing system 10includes an ultrasonic transmitter 20 and an ultrasonic receiver 30under a platen 40. The ultrasonic transmitter 20 may include asubstantially planar piezoelectric transmitter layer 22 and may becapable of functioning as a plane wave generator. Ultrasonic waves maybe generated by applying a voltage to the piezoelectric layer to expandor contract the layer, depending upon the signal applied, therebygenerating a plane wave. In this example, the control system 50 may becapable of causing a voltage that may be applied to the piezoelectrictransmitter layer 22 via a first transmitter electrode 24 and a secondtransmitter electrode 26. In this fashion, an ultrasonic wave may bemade by changing the thickness of the layer via a piezoelectric effect.This ultrasonic wave may travel towards a finger (or other object to bedetected), passing through the platen 40. A portion of the wave notabsorbed or transmitted by the object to be detected may be reflected soas to pass back through the platen 40 and be received by the ultrasonicreceiver 30. The first and second transmitter electrodes 24 and 26 maybe metallized electrodes, for example, metal layers that coat opposingsides of the piezoelectric transmitter layer 22.

The ultrasonic receiver 30 may include an array of sensor pixel circuits32 disposed on a substrate 34, which also may be referred to as abackplane, and a piezoelectric receiver layer 36. In someimplementations, each sensor pixel circuit 32 may include one or moreTFT elements, electrical interconnect traces and, in someimplementations, one or more additional circuit elements such as diodes,capacitors, and the like. Each sensor pixel circuit 32 may be configuredto convert an electric charge generated in the piezoelectric receiverlayer 36 proximate to the pixel circuit into an electrical signal. Eachsensor pixel circuit 32 may include a pixel input electrode 38 thatelectrically couples the piezoelectric receiver layer 36 to the sensorpixel circuit 32.

In the illustrated implementation, a receiver bias electrode 39 isdisposed on a side of the piezoelectric receiver layer 36 proximal toplaten 40. The receiver bias electrode 39 may be a metallized electrodeand may be grounded or biased to control which signals may be passed tothe array of sensor pixel circuits 32. Ultrasonic energy that isreflected from the exposed (top) surface 42 of the platen 40 may beconverted into localized electrical charges by the piezoelectricreceiver layer 36. These localized charges may be collected by the pixelinput electrodes 38 and passed on to the underlying sensor pixelcircuits 32. The charges may be amplified by the sensor pixel circuits32 and provided to the control system 50.

The control system 50 may be electrically connected (directly orindirectly) with the first transmitter electrode 24 and the secondtransmitter electrode 26, as well as with the receiver bias electrode 39and the sensor pixel circuits 32 on the substrate 34. In someimplementations, the control system 50 may operate substantially asdescribed above. For example, the control system 50 may be capable ofprocessing the amplified signals received from the sensor pixel circuits32.

The control system 50 may be capable of controlling the ultrasonictransmitter 20 and/or the ultrasonic receiver 30 to obtain fingerprintimage information, e.g., by obtaining fingerprint images. Whether or notthe touch/fingerprint sensing system 10 includes an ultrasonictransmitter 20, the control system 50 may be capable of controllingaccess to one or more devices based, at least in part, on thefingerprint image information. The touch/fingerprint sensing system 10(or an associated device) may include a memory system that includes oneor more memory devices. In some implementations, the control system 50may include at least a portion of the memory system. The control system50 may be capable of capturing a fingerprint image and storingfingerprint image information in the memory system. In someimplementations, the control system 50 may be capable of capturing afingerprint image and storing fingerprint image information in thememory system even while maintaining the ultrasonic transmitter 20 in an“off” state.

In some implementations, the control system 50 may be capable ofoperating the touch/fingerprint sensing system in an ultrasonic imagingmode or a force-sensing mode. In some implementations, the controlsystem may be capable of maintaining the ultrasonic transmitter 20 in an“off” state when operating the touch/fingerprint sensing system in aforce-sensing mode. The ultrasonic receiver 30 may be capable offunctioning as a force sensor when the touch/fingerprint sensing system10 is operating in the force-sensing mode.

In some implementations, the control system 50 may be capable ofcontrolling other devices, such as a display system, a communicationsystem, etc. In some implementations, for example, the control system 50may be capable of powering on one or more components of a device such asthe display device 940, which is described below with reference to FIGS.9A and 9B. Accordingly, in some implementations the control system 50also may include one or more components similar to the processor 921,the array driver 922 and/or the driver controller 929 shown in FIG. 9B.In some implementations, the control system 50 may be capable ofdetecting a touch or tap received via the ultrasonic receiver 30 servingas a force-sensing device and activating at least one feature of themobile display device in response to the touch or tap. The “feature” maybe a component, a software application, etc.

The platen 40 can be any appropriate material that can be acousticallycoupled to the receiver, with examples including plastic, ceramic,sapphire, metal and glass. In some implementations, the platen 40 can bea cover plate, e.g., a cover glass or a lens glass for a display.Particularly when the ultrasonic transmitter 20 is in use, fingerprintdetection and imaging can be performed through relatively thick platensif desired, e.g., 3 mm and above. However, for implementations in whichthe ultrasonic receiver 30 is capable of imaging fingerprints in a forcedetection mode, a thinner and relatively more compliant platen 40 may bedesirable. According to some such implementations, the platen 40 mayinclude one or more polymers, such as one or more types of parylene, andmay be substantially thinner. In some such implementations, the platen40 may be tens of microns thick or even less than 10 microns thick.

Examples of piezoelectric materials that may be used to form thepiezoelectric receiver layer 36 include piezoelectric polymers havingappropriate acoustic properties, for example, an acoustic impedancebetween about 2.5 MRayls and 5 MRayls. Specific examples ofpiezoelectric materials that may be employed include ferroelectricpolymers such as polyvinylidene fluoride (PVDF) and polyvinylidenefluoride-trifluoroethylene (PVDF-TrFE) copolymers. Examples of PVDFcopolymers include 60:40 (molar percent) PVDF-TrFE, 70:30 PVDF-TrFE,80:20 PVDF-TrFE, and 90:10 PVDR-TrFE. Other examples of piezoelectricmaterials that may be employed include polyvinylidene chloride (PVDC)homopolymers and copolymers, polytetrafluoroethylene (PTFE) homopolymersand copolymers, and diisopropylammonium bromide (DIPAB).

The thickness of each of the piezoelectric transmitter layer 22 and thepiezoelectric receiver layer 36 may be selected so as to be suitable forgenerating and receiving ultrasonic waves. In one example, a PVDFpiezoelectric transmitter layer 22 is approximately 28 μm thick and aPVDF-TrFE receiver layer 36 is approximately 12 μm thick. Examplefrequencies of the ultrasonic waves may be in the range of 5 MHz to 30MHz, with wavelengths on the order of a millimeter or less.

FIG. 8B shows an exploded view of an alternative example of atouch/fingerprint sensing system. In this example, the piezoelectricreceiver layer 36 has been formed into discrete elements 37. In theimplementation shown in FIG. 8B, each of the discrete elements 37corresponds with a single pixel input electrode 38 and a single sensorpixel circuit 32. However, in alternative implementations of thetouch/fingerprint sensing system 10, there is not necessarily aone-to-one correspondence between each of the discrete elements 37, asingle pixel input electrode 38 and a single sensor pixel circuit 32.For example, in some implementations there may be multiple pixel inputelectrodes 38 and sensor pixel circuits 32 for a single discrete element37.

FIGS. 8A and 8B show example arrangements of ultrasonic transmitters andreceivers in a touch/fingerprint sensing system, with other arrangementspossible. For example, in some implementations, the ultrasonictransmitter 20 may be above the ultrasonic receiver 30 and thereforecloser to the object(s) 25 to be detected. In some implementations, thetouch/fingerprint sensing system 10 may include an acoustic delay layer.For example, an acoustic delay layer can be incorporated into thetouch/fingerprint sensing system 10 between the ultrasonic transmitter20 and the ultrasonic receiver 30. An acoustic delay layer can beemployed to adjust the ultrasonic pulse timing, and at the same timeelectrically insulate the ultrasonic receiver 30 from the ultrasonictransmitter 20. The acoustic delay layer may have a substantiallyuniform thickness, with the material used for the delay layer and/or thethickness of the delay layer selected to provide a desired delay in thetime for reflected ultrasonic energy to reach the ultrasonic receiver30. In doing so, the range of time during which an energy pulse thatcarries information about the object by virtue of having been reflectedby the object may be made to arrive at the ultrasonic receiver 30 duringa time range when it is unlikely that energy reflected from other partsof the touch/fingerprint sensing system 10 is arriving at the ultrasonicreceiver 30. In some implementations, the substrate 34 and/or the platen40 may serve as an acoustic delay layer.

FIGS. 9A and 9B show examples of system block diagrams illustrating adisplay device that includes a touch/fingerprint sensing system asdescribed herein. The display device 940 may be, for example, mobiledisplay device such as a smart phone, a cellular or mobile telephone,etc. However, the same components of the display device 940 or slightvariations thereof are also illustrative of various types of displaydevices such as televisions, computers, tablets, e-readers, hand-helddevices and portable media devices.

In this example, the display device 940 includes a housing 941, adisplay 930, a touch/fingerprint sensing system 10 (which may part of orseparated from the visual display 930), an antenna 943, a speaker 945,an input device 948 and a microphone 946. The housing 941 may be formedfrom any of a variety of manufacturing processes, including injectionmolding, and vacuum forming. In addition, the housing 941 may be madefrom any of a variety of materials, including, but not limited to:plastic, metal, glass, rubber and ceramic, or a combination thereof. Thehousing 941 may include removable portions (not shown) that may beinterchanged with other removable portions of different color, orcontaining different logos, pictures, or symbols.

The display 930 may be any of a variety of displays, including aflat-panel display, such as plasma, organic light-emitting diode (OLED)or liquid crystal display (LCD), or a non-flat-panel display, such as acathode ray tube (CRT) or other tube device. In addition, the display930 may include an interferometric modulator (IMOD)-based display or amicro-shutter based display.

The components of one example of the display device 940 areschematically illustrated in FIG. 9B. Here, the display device 940includes a housing 941 and may include additional components at leastpartially enclosed therein. For example, the display device 940 includesa network interface 927 that includes an antenna 943 which may becoupled to a transceiver 947. The network interface 927 may be a sourcefor image data that could be displayed on the display device 940.Accordingly, the network interface 927 is one example of an image sourcemodule, but the processor 921 and the input device 948 also may serve asan image source module. The transceiver 947 is connected to a processor921, which is connected to conditioning hardware 952. The conditioninghardware 952 may be capable of conditioning a signal (such as applying afilter or otherwise manipulating a signal). The conditioning hardware952 may be connected to a speaker 945 and a microphone 946. Theprocessor 921 also may be connected to an input device 948 and a drivercontroller 929. The driver controller 929 may be coupled to a framebuffer 928, and to an array driver 922, which in turn may be coupled toa display array 930. One or more elements in the display device 940,including elements not specifically depicted in FIG. 9B, may be capableof functioning as a memory device and be capable of communicating withthe processor 921 or other components of a control system. In someimplementations, a power supply 950 may provide power to substantiallyall components in the particular display device 940 design.

In this example, the display device 940 also includes a touch andfingerprint controller 977. The touch and fingerprint controller 977may, for example, be a part of a control system 50 or a control system104 such as that described above. Accordingly, in some implementationsthe touch and fingerprint controller 977 (and/or other components of thecontrol system 50) may include one or more memory devices. In someimplementations, the control system 50 also may include components suchas the processor 921, the array driver 922 and/or the driver controller929 shown in FIG. 9B. The touch and fingerprint controller 977 may becapable of communicating with the touch/fingerprint sensing system 10,e.g., via routing wires, and may be capable of controlling thetouch/fingerprint sensing system 10. The touch and fingerprintcontroller 977 may be capable of determining a location and/or movementof one or more objects, such as fingers, on or proximate thetouch/fingerprint sensing system 10. In alternative implementations,however, the processor 921 (or another part of the control system 50)may be capable of providing some or all of the functionality of thetouch and fingerprint controller 977, the control system 50 and/or ofthe control system 104.

The touch and fingerprint controller 977 (and/or another element of thecontrol system 50) may be capable of providing input for controlling thedisplay device 940 according to one or more touch locations. In someimplementations, the touch and fingerprint controller 977 may be capableof determining movements of one or more touch locations and of providinginput for controlling the display device 940 according to the movements.Alternatively, or additionally, the touch and fingerprint controller 977may be capable of determining locations and/or movements of objects thatare proximate the display device 940. Accordingly, the touch andfingerprint controller 977 may be capable of detecting finger or stylusmovements, hand gestures, etc., even if no contact is made with thedisplay device 40. The touch and fingerprint controller 977 may becapable of providing input for controlling the display device 40according to such detected movements and/or gestures.

As described elsewhere herein, the touch and fingerprint controller 977(or another element of the control system 50) may be capable ofproviding one or more fingerprint detection operational modes.Accordingly, in some implementations the touch and fingerprintcontroller 977 (or another element of the control system 50) may becapable of producing fingerprint images. In some implementations, suchas when an ultrasonic sensor array of the fingerprint sensor system isphysically separated from the visual display 930, the controller for thefingerprint sensor system may be separate from and operate largelyindependent of the touch controller.

In some implementations, the touch/fingerprint sensing system 10 mayinclude an ultrasonic receiver 30 and/or an ultrasonic transmitter 20such as described elsewhere herein. According to some suchimplementations, the touch and fingerprint controller 977 (or anotherelement of the control system 50) may be capable of receiving input fromthe ultrasonic receiver 30 and powering on or “waking up” the ultrasonictransmitter 20 and/or another component of the display device 940.

The network interface 927 includes the antenna 943 and the transceiver947 so that the display device 940 may communicate with one or moredevices over a network. The network interface 927 also may have someprocessing capabilities to relieve, for example, data processingrequirements of the processor 921. The antenna 943 may transmit andreceive signals. In some implementations, the antenna 943 transmits andreceives RF signals according to the IEEE 16.11 standard, including IEEE16.11(a), (b), or (g), or the IEEE 802.11 standard, including IEEE802.11a, b, g, n, and further implementations thereof. In some otherimplementations, the antenna 943 transmits and receives RF signalsaccording to the Bluetooth® standard. In the case of a cellulartelephone, the antenna 943 may be designed to receive code divisionmultiple access (CDMA), frequency division multiple access (FDMA), timedivision multiple access (TDMA), Global System for Mobile communications(GSM), GSM/General Packet Radio Service (GPRS), Enhanced Data GSMEnvironment (EDGE), Terrestrial Trunked Radio (TETRA), Wideband-CDMA(W-CDMA), Evolution Data Optimized (EV-DO), 1×EV-DO, EV-DO Rev A, EV-DORev B, High Speed Packet Access (HSPA), High Speed Downlink PacketAccess (HSDPA), High Speed Uplink Packet Access (HSUPA), Evolved HighSpeed Packet Access (HSPA+), Long Term Evolution (LTE), AMPS, or otherknown signals that are used to communicate within a wireless network,such as a system utilizing 3G, 4G or 5G technology. The transceiver 947may pre-process the signals received from the antenna 943 so that theymay be received by and further manipulated by the processor 921. Thetransceiver 947 also may process signals received from the processor 921so that they may be transmitted from the display device 940 via theantenna 943.

In some implementations, the transceiver 947 may be replaced by areceiver. In addition, in some implementations, the network interface927 may be replaced by an image source, which may store or generateimage data to be sent to the processor 921. The processor 921 maycontrol the overall operation of the display device 940. The processor921 receives data, such as compressed image data from the networkinterface 927 or an image source, and processes the data into raw imagedata or into a format that may be readily processed into raw image data.The processor 921 may send the processed data to the driver controller929 or to the frame buffer 928 for storage. Raw data typically refers tothe information that identifies the image characteristics at eachlocation within an image. For example, such image characteristics mayinclude color, saturation and gray-scale level.

The processor 921 may include a microcontroller, CPU, or logic unit tocontrol operation of the display device 940. The conditioning hardware952 may include amplifiers and filters for transmitting signals to thespeaker 945, and for receiving signals from the microphone 946. Theconditioning hardware 952 may be discrete components within the displaydevice 940, or may be incorporated within the processor 921 or othercomponents.

The driver controller 929 may take the raw image data generated by theprocessor 921 either directly from the processor 921 or from the framebuffer 928 and may re-format the raw image data appropriately for highspeed transmission to the array driver 922. In some implementations, thedriver controller 929 may re-format the raw image data into a data flowhaving a raster-like format, such that it has a time order suitable forscanning across the display array 930. Then the driver controller 929sends the formatted information to the array driver 922. Although adriver controller 929, such as an LCD controller, is often associatedwith the system processor 921 as a stand-alone Integrated Circuit (IC),such controllers may be implemented in many ways. For example,controllers may be embedded in the processor 921 as hardware, embeddedin the processor 921 as software, or fully integrated in hardware withthe array driver 922.

The array driver 922 may receive the formatted information from thedriver controller 929 and may re-format the video data into a parallelset of waveforms that are applied many times per second to the hundreds,and sometimes thousands (or more), of leads coming from the display'sx-y matrix of display elements.

In some implementations, the driver controller 929, the array driver922, and the display array 930 are appropriate for any of the types ofdisplays described herein. For example, the driver controller 929 may bea conventional display controller or a bi-stable display controller(such as an IMOD display element controller). Additionally, the arraydriver 922 may be a conventional driver or a bi-stable display driver.Moreover, the display array 930 may be a conventional display array or abi-stable display. In some implementations, the driver controller 929may be integrated with the array driver 922. Such an implementation maybe useful in highly integrated systems, for example, mobile phones,portable-electronic devices, watches or small-area displays.

In some implementations, the input device 948 may be capable ofallowing, for example, a user to control the operation of the displaydevice 940. The input device 948 may include a keypad, such as a QWERTYkeyboard or a telephone keypad, a button, a switch, a rocker, atouch-sensitive screen, a touch-sensitive screen integrated with thedisplay array 930, or a pressure- or heat-sensitive membrane. Themicrophone 946 may be capable of functioning as an input device for thedisplay device 940. In some implementations, voice commands through themicrophone 946 may be used for controlling operations of the displaydevice 940.

The power supply 950 may include a variety of energy storage devices.For example, the power supply 950 may be a rechargeable battery, such asa nickel-cadmium battery or a lithium-ion battery. In implementationsusing a rechargeable battery, the rechargeable battery may be chargeableusing power coming from, for example, a wall socket or a photovoltaicdevice or array. Alternatively, the rechargeable battery may bewirelessly chargeable. The power supply 950 also may be a renewableenergy source, a capacitor, or a solar cell, including a plastic solarcell or solar-cell paint. The power supply 950 also may be capable ofreceiving power from a wall outlet.

In some implementations, control programmability resides in the drivercontroller 929 which may be located in several places in the electronicdisplay system. In some other implementations, control programmabilityresides in the array driver 922. The above-described optimization may beimplemented in any number of hardware and/or software components and invarious configurations.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The various illustrative logics, logical blocks, modules, circuits andalgorithm processes described in connection with the implementationsdisclosed herein may be implemented as electronic hardware, computersoftware, or combinations of both. The interchangeability of hardwareand software has been described generally, in terms of functionality,and illustrated in the various illustrative components, blocks, modules,circuits and processes described above. Whether such functionality isimplemented in hardware or software depends upon the particularapplication and design constraints imposed on the overall system.

The hardware and data processing apparatus used to implement the variousillustrative logics, logical blocks, modules and circuits described inconnection with the aspects disclosed herein may be implemented orperformed with a general purpose single- or multi-chip processor, adigital signal processor (DSP), an application specific integratedcircuit (ASIC), a field programmable gate array (FPGA) or otherprogrammable logic device, discrete gate or transistor logic, discretehardware components, or any combination thereof designed to perform thefunctions described herein. A general purpose processor may be amicroprocessor, or, any conventional processor, controller,microcontroller, or state machine. A processor also may be implementedas a combination of computing devices, e.g., a combination of a DSP anda microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration. In some implementations, particular processes and methodsmay be performed by circuitry that is specific to a given function.

In one or more aspects, the functions described may be implemented inhardware, digital electronic circuitry, computer software, firmware,including the structures disclosed in this specification and theirstructural equivalents thereof, or in any combination thereof.Implementations of the subject matter described in this specificationalso may be implemented as one or more computer programs, i.e., one ormore modules of computer program instructions, encoded on a computerstorage media for execution by, or to control the operation of, dataprocessing apparatus.

If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on acomputer-readable medium, such as a non-transitory medium. The processesof a method or algorithm disclosed herein may be implemented in aprocessor-executable software module which may reside on acomputer-readable medium. Computer-readable media include both computerstorage media and communication media including any medium that may beenabled to transfer a computer program from one place to another.Storage media may be any available media that may be accessed by acomputer. By way of example, and not limitation, non-transitory mediamay include RAM, ROM, EEPROM, CD-ROM or other optical disk storage,magnetic disk storage or other magnetic storage devices, or any othermedium that may be used to store desired program code in the form ofinstructions or data structures and that may be accessed by a computer.Also, any connection may be properly termed a computer-readable medium.Disk and disc, as used herein, includes compact disc (CD), laser disc,optical disc, digital versatile disc (DVD), floppy disk, and blu-raydisc where disks usually reproduce data magnetically, while discsreproduce data optically with lasers. Combinations of the above shouldalso be included within the scope of computer-readable media.Additionally, the operations of a method or algorithm may reside as oneor any combination or set of codes and instructions on a machinereadable medium and computer-readable medium, which may be incorporatedinto a computer program product.

Various modifications to the implementations described in thisdisclosure may be readily apparent to those having ordinary skill in theart, and the generic principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the disclosure is not intended to be limited to theimplementations shown herein, but is to be accorded the widest scopeconsistent with the claims, the principles and the novel featuresdisclosed herein. The word “exemplary” is used exclusively herein, if atall, to mean “serving as an example, instance, or illustration.” Anyimplementation described herein as “exemplary” is not necessarily to beconstrued as preferred or advantageous over other implementations.

Certain features that are described in this specification in the contextof separate implementations also may be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also may be implemented inmultiple implementations separately or in any suitable subcombination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination may in some cases be excised from thecombination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemsmay generally be integrated together in a single software product orpackaged into multiple software products. Additionally, otherimplementations are within the scope of the following claims. In somecases, the actions recited in the claims may be performed in a differentorder and still achieve desirable results.

It will be understood that unless features in any of the particulardescribed implementations are expressly identified as incompatible withone another or the surrounding context implies that they are mutuallyexclusive and not readily combinable in a complementary and/orsupportive sense, the totality of this disclosure contemplates andenvisions that specific features of those complementary implementationsmay be selectively combined to provide one or more comprehensive, butslightly different, technical solutions. It will therefore be furtherappreciated that the above description has been given by way of exampleonly and that modifications in detail may be made within the scope ofthis disclosure.

What is claimed is:
 1. A fingerprint sensing apparatus, comprising: afingerprint sensor system; and a control system capable of: receivingfingerprint sensor data from the fingerprint sensor system; determiningfingerprint sensor data blocks for at least a portion of the fingerprintsensor data; calculating characteristics of variability for fingerprintsensor data corresponding to each of the fingerprint sensor data blocks;and determining, according to the characteristics of variability,whether an object is positioned proximate a portion of the fingerprintsensor system.
 2. The apparatus of claim 1, wherein the control systemis capable of providing an object output signal indicating whether anobject is positioned proximate a portion of the fingerprint sensorsystem.
 3. The apparatus of claim 1, wherein the control system isfurther capable of determining whether an object that is positionedproximate the portion of the fingerprint sensor system is a stylus. 4.The apparatus of claim 1, wherein the fingerprint sensor data blockscorrespond to blocks of fingerprint sensor pixels of the fingerprintsensor system.
 5. The apparatus of claim 4, wherein the blocks offingerprint sensor pixels are contiguous.
 6. The apparatus of claim 5,wherein the blocks of fingerprint sensor pixels include substantiallyall fingerprint sensor pixels of the fingerprint sensor system.
 7. Theapparatus of claim 4, wherein the blocks of fingerprint sensor pixelshave at least two different sizes.
 8. The apparatus of claim 1, whereinthe control system is further capable of applying one or more filters tothe fingerprint sensor data before calculating the characteristics ofvariability.
 9. The apparatus of claim 8, wherein the one or morefilters pass a range of spatial frequencies corresponding to fingerprintfeatures.
 10. The apparatus of claim 1, wherein the control system isfurther capable of determining whether an object is a finger.
 11. Theapparatus of claim 1, wherein the control system is further capable ofextracting fingerprint features from the fingerprint sensor data if thecontrol system determines that the object is a finger.
 12. The apparatusof claim 1, wherein the fingerprint sensor system includes an ultrasonicsensor array.
 13. The apparatus of claim 12, wherein the control systemis capable of assessing an acoustic impedance of one or more objectsproximate the fingerprint sensor system.
 14. The apparatus of claim 13,wherein the control system is capable of determining whether an objecthas an acoustic impedance that is within an acoustic impedance rangecorresponding to that of skin.
 15. The apparatus of claim 14, whereinthe acoustic impedance range is from 1.3 MRayls to 2.1 MRayls.
 16. Theapparatus of claim 14, wherein the control system is capable ofdetermining whether a skin-covered object is a finger or another bodypart.
 17. The apparatus of claim 1, wherein determining whether anobject is positioned proximate a portion of the fingerprint sensorsystem involves determining whether the characteristics of variabilityare above or below a threshold value.
 18. The apparatus of claim 1,wherein the characteristics of variability are based, at least in part,on a fingerprint sensor signal value or a fingerprint sensor signalgradient.
 19. The apparatus of claim 1, wherein the control system iscapable of pre-processing the fingerprint sensor data prior tocalculating the characteristics of variability.
 20. The apparatus ofclaim 1, wherein the control system is capable of pre-processing thefingerprint sensor data prior to determining whether an object ispositioned proximate a portion of the fingerprint sensor system.
 21. Theapparatus of claim 20, wherein the pre-processing involves at least oneprocess selected from a group of processes consisting of filtering, gaincompensation, offset adjustment, background subtraction, a de-noiseoperation, contrast enhancement, scaling, linearization and clipping.22. The apparatus of claim 20, wherein the pre-processing involves:identifying malfunctioning sensor pixels, the malfunctioning sensorpixels including at least one of dead sensor pixels or low-performingsensor pixels; and excluding fingerprint sensor data from themalfunctioning sensor pixels.
 23. A fingerprint sensing method,comprising: receiving fingerprint sensor data from a fingerprint sensorsystem; determining fingerprint sensor data blocks for at least aportion of the fingerprint sensor data; calculating characteristics ofvariability for fingerprint sensor data corresponding to each of thefingerprint sensor data blocks; and determining, according to thecharacteristics of variability, whether an object is positionedproximate a portion of the fingerprint sensor system.
 24. The method ofclaim 23, wherein the fingerprint sensor data blocks correspond toblocks of fingerprint sensor pixels.
 25. The method of claim 23, furthercomprising determining whether an object is a finger.
 26. The method ofclaim 23, further comprising assessing an acoustic impedance of one ormore objects proximate the fingerprint sensor system.
 27. Anon-transitory medium having software stored thereon, the softwareincluding instructions for controlling a fingerprint sensing apparatusto: receive fingerprint sensor data from a fingerprint sensor system;determine fingerprint sensor data blocks for at least a portion of thefingerprint sensor data; calculate characteristics of variability forfingerprint sensor data corresponding to each of the fingerprint sensordata blocks; and determine, according to the characteristics ofvariability, whether an object is positioned proximate a portion of thefingerprint sensor system.
 28. The non-transitory medium of claim 27,wherein the fingerprint sensor data blocks correspond to blocks offingerprint sensor pixels.
 29. The non-transitory medium of claim 27,wherein the software includes instructions for controlling thefingerprint sensing apparatus to determine whether an object is afinger.
 30. The non-transitory medium of claim 27, wherein the softwareincludes instructions for controlling the fingerprint sensing apparatusto assess an acoustic impedance of one or more objects proximate thefingerprint sensor system.
 31. A fingerprint sensing apparatus,comprising: a fingerprint sensor system; and control means for:receiving fingerprint sensor data from the fingerprint sensor system;determining fingerprint sensor data blocks for at least a portion of thefingerprint sensor data; calculating characteristics of variability forfingerprint sensor data corresponding to each of the fingerprint sensordata blocks; and determining, according to the characteristics ofvariability, whether an object is positioned proximate a portion of thefingerprint sensor system.
 32. The apparatus of claim 31, wherein thecontrol means includes means for determining whether an object that ispositioned proximate the portion of the fingerprint sensor system is astylus.
 33. The apparatus of claim 31, wherein the fingerprint sensordata blocks correspond to blocks of fingerprint sensor pixels of thefingerprint sensor system.
 34. The apparatus of claim 33, wherein theblocks of fingerprint sensor pixels are contiguous.